Nov 03 20:28:52.879628 ip-10-0-135-141 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:28:53.337298 ip-10-0-135-141 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:53.337298 ip-10-0-135-141 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:28:53.337298 ip-10-0-135-141 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:53.337298 ip-10-0-135-141 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:28:53.342990 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.342912    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.342990 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.342990    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.342994    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.342998    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343001    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343004    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343007    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343009    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343012    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343015    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343019    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343023    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343026    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343029    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343032    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343035    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343037    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343040    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343050    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343053    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.343060 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343056    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343062    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343064    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343068    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343071    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343073    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343076    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343078    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343081    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343083    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343087    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343089    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343091    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343094    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343097    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343099    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343102    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343104    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343108    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.343564 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343110    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343113    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343115    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343118    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343122    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343126    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343129    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343132    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343134    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343137    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343139    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343142    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343144    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343147    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343150    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343153    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343155    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343158    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343161    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343164    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.344036 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343168    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343171    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343173    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343176    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343179    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343181    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343184    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343186    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343189    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343191    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343193    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343197    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343199    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343202    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343204    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343207    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343209    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343212    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343215    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.344549 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343217    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343219    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343225    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343228    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343231    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343234    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343236    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343239    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343242    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343245    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343248    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343251    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343254    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343256    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343259    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343261    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343264    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343267    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343269    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.345023 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343272    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343274    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343277    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343279    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343282    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.343285    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344112    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344119    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344123    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344127    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344130    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344132    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344135    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344137    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344140    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344143    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344146    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344149    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344152    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344155    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.345517 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344157    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344159    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344162    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344165    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344168    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344171    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344173    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344176    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344178    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344181    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344183    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344186    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344188    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344191    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344194    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344196    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344199    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344201    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344214    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344218    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.346000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344220    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344224    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344227    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344230    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344233    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344236    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344239    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344241    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344244    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344247    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344249    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344252    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344255    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344257    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344260    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344263    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344265    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344268    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344270    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.346495 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344273    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344275    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344278    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344280    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344283    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344285    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344288    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344296    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344300    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344303    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344308    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344311    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344314    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344318    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344321    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344323    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344326    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344329    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344332    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.347073 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344349    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344351    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344354    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344356    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344359    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344362    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344364    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344367    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344370    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344373    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344375    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344378    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344380    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344383    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344385    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344388    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344390    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344393    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344395    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344398    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.347567 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344400    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344403    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344406    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344408    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344411    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344413    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344416    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344419    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344422    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344426    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344428    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344431    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344434    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344436    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344913    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344918    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344921    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344924    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344927    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344930    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.348058 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344933    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344935    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344938    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344940    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344943    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344946    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344948    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344950    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344953    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344955    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344958    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344960    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344963    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344965    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344968    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344970    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344973    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344975    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344979    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344982    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.348551 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344985    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344987    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344990    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344993    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344995    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.344998    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345001    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345003    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345006    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345008    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345011    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345013    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345016    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345018    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345021    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345024    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345026    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345029    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345032    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345035    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.349024 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345037    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345042    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345046    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345050    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345053    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345055    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345058    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345060    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345063    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345065    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345068    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345072    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345075    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345077    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345082    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345085    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345088    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345090    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.349506 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345093    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345095    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345098    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345100    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345103    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345106    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345112    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345115    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345118    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345120    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345123    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345125    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345128    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345131    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345133    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345136    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345138    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345141    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345143    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345146    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.349940 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345149    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345151    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345154    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345157    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345159    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345161    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345165    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345167    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345171    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345174    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345177    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345179    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345182    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345186    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345190    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345192    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345195    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345198    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345202    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.350435 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345204    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345207    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345210    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345324    2578 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345347    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345357    2578 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345362    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345367    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345370    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345375    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345379    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345382    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345385    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345389    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345392    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345396    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345400    2578 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345403    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345406    2578 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345409    2578 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345413    2578 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345419    2578 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345422    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345425    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:28:53.350883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345428    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345432    2578 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345435    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345439    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345442    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345445    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345448    2578 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345452    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345456    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345459    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345462    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345466    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345469    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345473    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345476    2578 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345479    2578 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345482    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345486    2578 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345489    2578 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345492    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345496    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345499    2578 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345503    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345506    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345509    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:28:53.351467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345513    2578 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345516    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345519    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345522    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345527    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345531    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345534    2578 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345537    2578 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345544    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345547    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345550    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345554    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345557    2578 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345560    2578 flags.go:64] FLAG: --help="false"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345564    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345567    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345570    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345573    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345576    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345580    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345583    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345586    2578 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345590    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:28:53.352149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345593    2578 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345596    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345600    2578 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345603    2578 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345606    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345609    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345612    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345614    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345617    2578 flags.go:64] FLAG: --lock-file=""
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345620    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345623    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345626    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345632    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345635    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345639    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345642    2578 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345645    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345649    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345652    2578 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345655    2578 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345659    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345662    2578 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345666    2578 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345669    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345672    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:28:53.352713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345675    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345678    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345681    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345684    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345687    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345694    2578 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345697    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345700    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345703    2578 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345706    2578 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345709    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345712    2578 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345714    2578 flags.go:64] FLAG: --port="10250"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345717    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345720    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f6e8820218a2a23b"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345723    2578 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345726    2578 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345729    2578 flags.go:64] FLAG: --register-node="true"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345732    2578 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345736    2578 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345739    2578 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345742    2578 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345746    2578 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345750    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345753    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345756    2578 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:28:53.353310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345759    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345762    2578 flags.go:64] FLAG: --runonce="false"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345765    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345768    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345771    2578 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345774    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345777    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345780    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345784    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345787    2578 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345790    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345793    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345796    2578 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345798    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345801    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345805    2578 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345807    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345813    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345815    2578 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345818    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345822    2578 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345825    2578 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345828    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345831    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345834    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:28:53.353947 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345837    2578 flags.go:64] FLAG: --v="2"
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345841    2578 flags.go:64] FLAG: --version="false"
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345846    2578 flags.go:64] FLAG: --vmodule=""
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345850    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.345854    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345986    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345990    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345993    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345995    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.345998    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346001    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346004    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346008    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346010    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346014    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346017    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346019    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346022    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346025    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346028    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.354533 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346030    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346034    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346038    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346041    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346043    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346046    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346048    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346051    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346054    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346056    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346061    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346064    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346067    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346070    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346073    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346076    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346079    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346082    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.355014 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346084    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346087    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346089    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346092    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346094    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346097    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346099    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346102    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346104    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346108    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346110    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346113    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346115    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346117    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346120    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346122    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346125    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346127    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346130    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.355498 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346134    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346138    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346141    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346143    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346146    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346150    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346153    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346156    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346160    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346163    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346166    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346168    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346171    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346174    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346177    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346179    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346182    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346184    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346187    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346189    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.355986 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346192    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346195    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346197    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346200    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346202    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346205    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346207    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346210    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346212    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346215    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346217    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346220    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346225    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346227    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346230    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346233    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346237    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346239    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346242    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.356630 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346246    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346249    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346251    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346254    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346256    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346260    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346262    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346265    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346267    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346270    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346272    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346275    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346278    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.346303    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346351    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346356    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.357437 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346359    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346362    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346364    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346367    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346369    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346372    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346375    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346377    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346380    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346383    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346385    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346388    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346390    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346393    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346397    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346400    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346404    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346406    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346409    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346412    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.358065 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346414    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346417    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346420    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346423    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346426    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346428    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346431    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346434    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346437    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346439    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346442    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346444    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346447    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346449    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346452    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346455    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346457    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346460    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346462    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346465    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.358860 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346467    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346470    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346472    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346475    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346478    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346480    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346484    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346487    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346492    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346494    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346498    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346500    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346504    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346508    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346511    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346515    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346518    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346520    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346523    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.359356 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346526    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346528    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346531    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346533    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346536    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346538    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346541    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346543    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346546    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346549    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346551    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346554    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346556    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346559    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346562    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346564    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346567    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346569    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346572    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.359977 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346576    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346578    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346583    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346585    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346588    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346590    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346593    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346596    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346598    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346601    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346604    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346609    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346612    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346615    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346618    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346620    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346623    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346626    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346628    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.360463 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346631    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346633    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346636    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346638    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346641    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346643    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.346646    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.353832    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.353848    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.354811    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.354817    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354892    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354898    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354902    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354905    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354908    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354911    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354914    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354917    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.360966 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354919    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354922    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354925    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354928    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354930    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354933    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354936    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354938    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354941    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354944    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354946    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354949    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354952    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354954    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354957    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354960    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354962    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354965    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354967    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354970    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.361446 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354972    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354977    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354980    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354982    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354985    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354989    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354993    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354996    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.354999    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355001    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355004    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355007    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355009    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355012    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355015    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355018    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355020    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355023    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355026    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.362000 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355029    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355032    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355034    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355037    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355039    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355042    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355044    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355047    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355049    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355052    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355054    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355057    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355063    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355066    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355068    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355071    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355074    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355076    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355079    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.362467 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355082    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355084    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355087    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355089    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355093    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355097    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355100    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355103    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355107    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355110    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355113    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355116    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355118    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355121    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355124    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355127    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355130    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355132    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355135    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.363220 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355137    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355139    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355142    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355144    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355147    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355149    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355152    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355155    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355158    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355161    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355164    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355169    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355171    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355175    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355177    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355180    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355183    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355185    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.363927 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355188    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.355217    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355407    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355413    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355415    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355418    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355421    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355423    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355426    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355428    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355431    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355433    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355436    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355439    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355441    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.364403 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355444    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355446    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355449    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355451    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355454    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355457    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355460    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355462    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355466    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355469    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355471    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355474    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355477    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355479    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355482    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355485    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355487    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355490    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355492    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.364797 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355495    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355497    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355501    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355505    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355511    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355514    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355517    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355519    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355522    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355525    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355527    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355530    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355532    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355535    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355537    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355540    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355542    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355545    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355548    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355550    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.365241 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355553    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355556    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355558    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355561    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355564    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355566    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355569    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355572    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355574    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355577    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355579    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355582    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355584    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355587    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355589    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355592    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355594    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355597    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355599    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355602    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.365719 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355605    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355607    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355609    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355612    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355615    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355617    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355620    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355622    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355625    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355628    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355631    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355633    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355635    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355638    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355640    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355644    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355648    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355651    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355655    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.366214 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355658    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355660    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355663    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355666    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355669    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355671    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355674    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355676    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355679    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355681    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355684    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355686    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355689    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355691    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355694    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.355721    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:28:53.366991 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355757    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355761    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355764    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355768    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355771    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355773    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355776    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355779    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355782    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355784    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355787    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355789    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355792    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355795    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355798    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355800    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355803    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355805    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355808    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355810    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:28:53.367668 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355812    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355815    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355818    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355820    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355823    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355825    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355828    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355830    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355833    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355835    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355838    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355841    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355843    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355847    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355851    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355854    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355857    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355860    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355863    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:28:53.368144 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355865    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355868    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355870    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355873    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355875    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355878    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355880    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355883    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355886    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355889    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355891    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355894    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355898    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355901    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355903    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355906    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355908    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355910    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355913    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:28:53.368792 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355916    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355918    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355920    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355923    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355925    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355928    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355930    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355933    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355935    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355938    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355941    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355943    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355946    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355948    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355950    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355953    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355957    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355960    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355963    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:28:53.369235 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355965    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355968    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355971    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355973    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355976    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355978    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355981    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355983    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355987    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355991    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355995    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.355998    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356001    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356004    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356006    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356008    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356011    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356013    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356016    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356019    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:28:53.369722 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356021    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356025    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356028    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356030    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356034    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356037    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356039    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356042    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:53.356044    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.356103    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.359551    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.360757    2578 server.go:1014] "Starting client certificate rotation"
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.360854    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.360889    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.363358    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:28:53.370224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.367641    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:28:53.379238 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.379224    2578 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:28:53.382941 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.382926    2578 log.go:25] "Validated CRI v1 image API"
Nov 03 20:28:53.383776 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.383745    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:28:53.388681 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.388657    2578 fs.go:135] Filesystem UUIDs: map[70a88080-6513-41d6-849b-2122546f9e20:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a734207b-1850-4683-8a8a-f08da5fea8f9:/dev/nvme0n1p4]
Nov 03 20:28:53.388759 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.388679    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:28:53.394030 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.393908    2578 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:53.392541105 +0000 UTC m=+0.375762581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095757 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec207adaec32ed2e14dff9b23eff26b7 SystemUUID:ec207ada-ec32-ed2e-14df-f9b23eff26b7 BootID:f4d7bd73-fca4-48d2-8ef9-04dea88c8ee6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f5:46:59:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f5:46:59:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:72:26:f4:d4:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:28:53.394030 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.394022    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:28:53.394198 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.394178    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:28:53.397070 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.397047    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:28:53.397214 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.397071    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-141.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:28:53.397258 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.397217    2578 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:28:53.397258 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.397225    2578 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:28:53.397307 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.397263    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:28:53.397307 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.397280    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:28:53.398239 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.398229    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:28:53.402271 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.402259    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:28:53.402398 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.402389    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:28:53.403265 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.403256    2578 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:28:53.403315 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.403269    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:28:53.403315 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.403280    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:28:53.403315 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.403289    2578 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:28:53.403315 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.403305    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:28:53.403655 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.403644    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:28:53.408836 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.408816    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:28:53.414366 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.414351    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:28:53.415275 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415263    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:28:53.415326 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415293    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:28:53.415473 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415463    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:28:53.415503 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415479    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:28:53.415503 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415486    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:28:53.415503 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415491    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:28:53.415503 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415497    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:28:53.415601 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.415504    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:28:53.416171 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.416161    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:28:53.416171 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.416171    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:28:53.416255 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.416177    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:28:53.416255 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.416183    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:28:53.416255 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.416192    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:28:53.416255 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.416200    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:28:53.417396 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.417385    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:28:53.417396 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.417394    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:28:53.421470 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.421457    2578 server.go:1263] "Started kubelet"
Nov 03 20:28:53.421617 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.421588    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:28:53.421682 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.421635    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:28:53.421724 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.421709    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:28:53.422706 ip-10-0-135-141 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:28:53.423000 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.422983    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:28:53.423901 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.423882    2578 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:28:53.428953 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.428927    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:28:53.429035 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.428975    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:28:53.429383 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.429366    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:28:53.429519 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.429496    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.430190 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430172    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:28:53.430261 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430245    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:28:53.430261 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430259    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:28:53.430391 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430377    2578 factory.go:55] Registering systemd factory
Nov 03 20:28:53.430439 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430400    2578 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:28:53.430439 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430410    2578 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:28:53.430439 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430423    2578 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:28:53.430606 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430595    2578 factory.go:153] Registering CRI-O factory
Nov 03 20:28:53.430606 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430606    2578 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:28:53.430703 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430689    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:28:53.430737 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430710    2578 factory.go:103] Registering Raw factory
Nov 03 20:28:53.430737 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.430720    2578 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:28:53.431672 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.431661    2578 manager.go:319] Starting recovery of all containers
Nov 03 20:28:53.443393 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.441652    2578 manager.go:324] Recovery completed
Nov 03 20:28:53.448433 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.448416    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.450135 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.450111    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.450182 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.450156    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.450182 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.450174    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.451633 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.451618    2578 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:28:53.451695 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.451633    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:28:53.451695 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.451654    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:28:53.451695 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.451655    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.452860 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.452846    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.452928 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.452871    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.452928 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.452882    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.453128 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.453117    2578 policy_none.go:50] "Start"
Nov 03 20:28:53.453170 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.453133    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:28:53.453170 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.453155    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:28:53.454260 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.454251    2578 policy_none.go:44] "Start"
Nov 03 20:28:53.478908 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.478880    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:28:53.481730 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.481716    2578 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:28:53.481823 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.481750    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:28:53.481823 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.481760    2578 server.go:87] "Starting device plugin registration server"
Nov 03 20:28:53.481994 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.481980    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:28:53.482054 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.481994    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:28:53.482104 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.482086    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:28:53.482157 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.482152    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:28:53.482209 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.482160    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:28:53.482713 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.482690    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:28:53.482790 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.482745    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.560448 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.560418    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-141.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:28:53.560448 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.560426    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-141.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:28:53.561706 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.560434    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-141.us-west-2.compute.internal.18e4a2f704cc6b56  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:53.421427542 +0000 UTC m=+0.404649018,LastTimestamp:2026-11-03 20:28:53.421427542 +0000 UTC m=+0.404649018,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-141.us-west-2.compute.internal,}"
Nov 03 20:28:53.578501 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.578472    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:28:53.579636 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.579618    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:28:53.579705 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.579644    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:28:53.579705 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.579668    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:28:53.579800 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.579705    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 20:28:53.582116 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.582101    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.582929 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.582908    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.583020 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.582940    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.583020 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.582953    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.583020 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.582974    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.618407 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.618277    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-141.us-west-2.compute.internal.18e4a2f706827e10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-141.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:53.450137104 +0000 UTC m=+0.433358583,LastTimestamp:2026-11-03 20:28:53.450137104 +0000 UTC m=+0.433358583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-141.us-west-2.compute.internal,}"
Nov 03 20:28:53.619073 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.619050    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 20:28:53.619131 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.619050    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-141.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 20:28:53.635346 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.635323    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:28:53.635412 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.635392    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jgb5"
Nov 03 20:28:53.679969 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.679949    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"]
Nov 03 20:28:53.680043 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.680005    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.680944 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.680930    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.681003 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.680958    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.681003 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.680968    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.682472 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.682461    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.682620 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.682605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.682678 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.682639    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.683138 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.683124    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.683138 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.683132    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.683241 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.683153    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.683241 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.683164    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.683241 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.683154    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.683241 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.683240    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.684523 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.684507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.684566 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.684533    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:53.685169 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.685152    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:53.685250 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.685173    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:53.685250 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.685187    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:53.711255 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.711234    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.715119 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.715101    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.731961 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.731941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.732037 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.731964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.732037 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.731979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc5d4efd8a63d505e3fa8dc27ffb1de-config\") pod \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ebc5d4efd8a63d505e3fa8dc27ffb1de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.830615 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.830579    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:53.832723 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.832785 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.832785 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832738    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:53.832785 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc5d4efd8a63d505e3fa8dc27ffb1de-config\") pod \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ebc5d4efd8a63d505e3fa8dc27ffb1de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.832899 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.832899 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832825    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc5d4efd8a63d505e3fa8dc27ffb1de-config\") pod \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ebc5d4efd8a63d505e3fa8dc27ffb1de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.832899 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.832844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.833483 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:53.833470    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:53.833521 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.833489    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-141.us-west-2.compute.internal\": node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:53.948260 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:53.948184    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.015974 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.015951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:54.019030 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.019010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:54.048440 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.048288    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.149710 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.149677    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.250337 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.250238    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.351349 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.351314    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.361332 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.361312    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:28:54.361482 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.361457    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:28:54.361545 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.361459    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:28:54.361545 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.361459    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a37db1c6ee22f4efb81f40e9e9d69904-f845c31920a170ae.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-135-141.us-west-2.compute.internal.18e4a2f70683276c\": read tcp 10.0.135.141:40396->3.19.45.153:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-135-141.us-west-2.compute.internal.18e4a2f70683276c  default   5215 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-141.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:53 +0000 UTC,LastTimestamp:2026-11-03 20:28:53.582956705 +0000 UTC m=+0.566178182,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-141.us-west-2.compute.internal,}"
Nov 03 20:28:54.452017 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.451978    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.553377 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.553296    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.640010 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.639966    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:54 +0000 UTC" deadline="2028-06-28 23:19:39.89719833 +0000 UTC"
Nov 03 20:28:54.640010 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.640001    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14474h50m45.257201417s"
Nov 03 20:28:54.654152 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.654127    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.710068 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.710041    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:28:54.711063 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.711046    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:28:54.711160 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.711076    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:28:54.711160 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:54.711087    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:28:54.755075 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.755036    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.855973 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.855901    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:54.957519 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:54.957482    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:55.057620 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.057581    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:28:55.132147 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.132059    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:55.153731 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.153708    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:55.231600 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.231566    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:55.260176 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.260122    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba15cf602964f30d9df06d36baaae3a5.slice/crio-c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9 WatchSource:0}: Error finding container c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9: Status 404 returned error can't find the container with id c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9
Nov 03 20:28:55.264536 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.264513    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc5d4efd8a63d505e3fa8dc27ffb1de.slice/crio-e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a WatchSource:0}: Error finding container e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a: Status 404 returned error can't find the container with id e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a
Nov 03 20:28:55.265641 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.265616    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:28:55.302427 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.302407    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:28:55.303594 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.303579    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:28:55.366631 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.366609    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:28:55.388745 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.388693    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:55.406161 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.406138    2578 apiserver.go:52] "Watching apiserver"
Nov 03 20:28:55.432290 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.432265    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:28:55.573116 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.573087    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:28:55.573433 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.573412    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-j68vs","kube-system/global-pull-secret-syncer-nl7nq","openshift-cluster-node-tuning-operator/tuned-vmcgc","openshift-dns/node-resolver-hk7dd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-ms89f","kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf","openshift-image-registry/node-ca-kxpt4","openshift-network-diagnostics/network-check-target-d82lk","kube-system/konnectivity-agent-9k7ks","openshift-multus/multus-additional-cni-plugins-zpntw","openshift-multus/multus-knqnr","openshift-network-operator/iptables-alerter-7dw2d"]
Nov 03 20:28:55.577370 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.577337    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:55.577469 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.577441    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:28:55.579498 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.579481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.581770 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.581750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.583977 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.583957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.586321 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.586299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.588667 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.588647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:55.588766 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.588702    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:28:55.590927 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.590910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.593271 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.593253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.595752 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.595736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.595845 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.595811    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:28:55.599040 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.599021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.601704 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.601683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.604411 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.604389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.608930 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.608890    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ebc5d4efd8a63d505e3fa8dc27ffb1de","Type":"ContainerStarted","Data":"e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a"}
Nov 03 20:28:55.609026 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.608941    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ba15cf602964f30d9df06d36baaae3a5","Type":"ContainerStarted","Data":"c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9"}
Nov 03 20:28:55.630995 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.630975    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:28:55.640393 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640345    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:28:55.640531 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640514    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.640581 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640547    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.640630 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640586    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.640678 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640666    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.640763 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640748    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bb4f5\""
Nov 03 20:28:55.640763 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640758    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:28:55.640849 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640765    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dfjfq\""
Nov 03 20:28:55.640849 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.640763    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kkpg\""
Nov 03 20:28:55.641880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.641865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-kubernetes\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.641931 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.641890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-var-lib-kubelet\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.641931 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.641906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-device-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.641931 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.641924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bce3daf-4de1-4fea-89e8-cd891e432355-konnectivity-ca\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.642079 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.641942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-bin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642079 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.641980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-etc-kubernetes\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642079 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysconfig\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-conf\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-lib-modules\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-host\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdbx\" (UniqueName: \"kubernetes.io/projected/42f1ee76-96ba-4983-b33a-b7321b678045-kube-api-access-6bdbx\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-conf-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-ovn\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-node-log\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-systemd\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-etc-tuned\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642462 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-kubelet\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-netns\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-netd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzmkw\" (UniqueName: \"kubernetes.io/projected/8cdac693-b083-4a91-bd5c-ea56d23bd771-kube-api-access-qzmkw\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-sys-fs\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-config\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-system-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-bin\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-kubelet-config\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-dbus\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-hostroot\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.642789 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-systemd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjr6x\" (UniqueName: \"kubernetes.io/projected/a641934f-d459-46af-be40-6aee398f3b82-kube-api-access-vjr6x\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x82\" (UniqueName: \"kubernetes.io/projected/c7871e65-e639-4c12-ad40-4a2dd1b90687-kube-api-access-82x82\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovn-node-metrics-cert\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7871e65-e639-4c12-ad40-4a2dd1b90687-iptables-alerter-script\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-kubelet\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.642979    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-modprobe-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-sys\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-socket-dir-parent\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-k8s-cni-cncf-io\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-systemd-units\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a641934f-d459-46af-be40-6aee398f3b82-serviceca\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643214    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-registration-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.643310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643241    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8tnq\""
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-cnibin\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643256    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgf4\" (UniqueName: \"kubernetes.io/projected/2c38cada-a1d9-40a5-83c2-ac7157fb9975-kube-api-access-jqgf4\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-var-lib-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbtk\" (UniqueName: \"kubernetes.io/projected/19efb517-1936-48f3-ab42-bd7ee7a87239-kube-api-access-tpbtk\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-system-cni-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-daemon-config\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-multus-certs\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bce3daf-4de1-4fea-89e8-cd891e432355-agent-certs\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-hosts-file\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cnibin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-etc-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlbz\" (UniqueName: \"kubernetes.io/projected/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-kube-api-access-jxlbz\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-log-socket\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.643893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-tmp\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a641934f-d459-46af-be40-6aee398f3b82-host\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7871e65-e639-4c12-ad40-4a2dd1b90687-host-slash\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-netns\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-slash\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-env-overrides\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-run\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-socket-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wql\" (UniqueName: \"kubernetes.io/projected/17928b13-2c16-48bd-9cad-e8920d246891-kube-api-access-j2wql\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-tmp-dir\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgxb\" (UniqueName: \"kubernetes.io/projected/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-kube-api-access-6rgxb\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.643999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-multus\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.644492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.644057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-script-lib\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.644989 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.644104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cni-binary-copy\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.647446 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.647407    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.647517 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.647489    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:28:55.647517 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.647496    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qtf8\""
Nov 03 20:28:55.647676 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.647659    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.744868 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-node-log\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.744979 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-systemd\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.744979 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-etc-tuned\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.744979 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.744979 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-node-log\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.744979 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.744983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-systemd\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-kubelet\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-netns\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745110    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745123    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-netns\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-kubelet\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-netd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmkw\" (UniqueName: \"kubernetes.io/projected/8cdac693-b083-4a91-bd5c-ea56d23bd771-kube-api-access-qzmkw\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-sys-fs\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-netd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-sys-fs\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745256    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-config\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-system-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-bin\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-kubelet-config\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.745435    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745446    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-bin\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745451    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-system-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745467    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-kubelet-config\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-dbus\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.745517    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.245482037 +0000 UTC m=+3.228703529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-dbus\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-hostroot\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-systemd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.745903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745642    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-hostroot\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745676    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-systemd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjr6x\" (UniqueName: \"kubernetes.io/projected/a641934f-d459-46af-be40-6aee398f3b82-kube-api-access-vjr6x\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x82\" (UniqueName: \"kubernetes.io/projected/c7871e65-e639-4c12-ad40-4a2dd1b90687-kube-api-access-82x82\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovn-node-metrics-cert\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7871e65-e639-4c12-ad40-4a2dd1b90687-iptables-alerter-script\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-kubelet\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-modprobe-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-kubelet\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-sys\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-socket-dir-parent\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-k8s-cni-cncf-io\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-modprobe-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-sys\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-socket-dir-parent\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.746751 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-k8s-cni-cncf-io\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.745998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-systemd-units\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-systemd-units\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a641934f-d459-46af-be40-6aee398f3b82-serviceca\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-registration-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-cnibin\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgf4\" (UniqueName: \"kubernetes.io/projected/2c38cada-a1d9-40a5-83c2-ac7157fb9975-kube-api-access-jqgf4\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-var-lib-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746176    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-registration-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbtk\" (UniqueName: \"kubernetes.io/projected/19efb517-1936-48f3-ab42-bd7ee7a87239-kube-api-access-tpbtk\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-cnibin\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746277    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-var-lib-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7871e65-e639-4c12-ad40-4a2dd1b90687-iptables-alerter-script\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-system-cni-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.747579 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-system-cni-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-daemon-config\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-multus-certs\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-multus-certs\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bce3daf-4de1-4fea-89e8-cd891e432355-agent-certs\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-hosts-file\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cnibin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-etc-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-hosts-file\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a641934f-d459-46af-be40-6aee398f3b82-serviceca\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746589    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cnibin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlbz\" (UniqueName: \"kubernetes.io/projected/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-kube-api-access-jxlbz\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-etc-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-log-socket\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:55.748466 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-log-socket\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-tmp\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a641934f-d459-46af-be40-6aee398f3b82-host\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7871e65-e639-4c12-ad40-4a2dd1b90687-host-slash\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746825    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-netns\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746856    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746864    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7871e65-e639-4c12-ad40-4a2dd1b90687-host-slash\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746876    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-netns\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-slash\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-slash\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.746850    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-env-overrides\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-run\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749339 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.746967    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.246951535 +0000 UTC m=+3.230173005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-run\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-socket-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wql\" (UniqueName: \"kubernetes.io/projected/17928b13-2c16-48bd-9cad-e8920d246891-kube-api-access-j2wql\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-tmp-dir\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgxb\" (UniqueName: \"kubernetes.io/projected/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-kube-api-access-6rgxb\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-socket-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-multus\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-multus\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-script-lib\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cni-binary-copy\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-kubernetes\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-var-lib-kubelet\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-device-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bce3daf-4de1-4fea-89e8-cd891e432355-konnectivity-ca\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-bin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747306    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-var-lib-kubelet\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.749880 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-etc-kubernetes\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysconfig\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747353    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-kubernetes\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-conf\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-tmp-dir\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-lib-modules\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747431    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-etc-kubernetes\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-host\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-bin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdbx\" (UniqueName: \"kubernetes.io/projected/42f1ee76-96ba-4983-b33a-b7321b678045-kube-api-access-6bdbx\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-conf-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysconfig\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-ovn\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-lib-modules\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-host\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-ovn\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747607    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-device-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747648    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-conf-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.750359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.746785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a641934f-d459-46af-be40-6aee398f3b82-host\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.750884 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747720    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-conf\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750884 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.747822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bce3daf-4de1-4fea-89e8-cd891e432355-konnectivity-ca\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.750884 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.749414    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-etc-tuned\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750884 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.749462    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-tmp\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.750884 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.749583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bce3daf-4de1-4fea-89e8-cd891e432355-agent-certs\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.753010 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.752986    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:28:55.753131 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753033    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:28:55.753131 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753072    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.753131 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753077    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.753131 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753074    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s87xv"
Nov 03 20:28:55.753131 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753123    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:28:55.753390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753142    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.753390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753192    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:28:55.753390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753195    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:28:55.753390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753198    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:28:55.753390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753247    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753402    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753452    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753453    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753499    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ff7gv\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753513    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753552    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753595    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7m98\""
Nov 03 20:28:55.753658 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753650    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:28:55.753911 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753868    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7b8b\""
Nov 03 20:28:55.753953 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.753921    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c6v9\""
Nov 03 20:28:55.756141 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.756124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-config\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.756141 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.756135    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.756490 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.756474    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.757615 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.757567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.758021 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.757986    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-daemon-config\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.758210 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.758189    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-env-overrides\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.758300 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.758281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cni-binary-copy\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.758450 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.758430    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-script-lib\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.760713 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.760690    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovn-node-metrics-cert\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.809658 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.809635    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:55.809766 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.809662    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:55.809766 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.809678    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:55.809766 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:55.809746    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.30972671 +0000 UTC m=+3.292948193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:55.812151 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.812125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjr6x\" (UniqueName: \"kubernetes.io/projected/a641934f-d459-46af-be40-6aee398f3b82-kube-api-access-vjr6x\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.812282 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.812125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x82\" (UniqueName: \"kubernetes.io/projected/c7871e65-e639-4c12-ad40-4a2dd1b90687-kube-api-access-82x82\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.812354 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.812322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgf4\" (UniqueName: \"kubernetes.io/projected/2c38cada-a1d9-40a5-83c2-ac7157fb9975-kube-api-access-jqgf4\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.812442 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.812322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbtk\" (UniqueName: \"kubernetes.io/projected/19efb517-1936-48f3-ab42-bd7ee7a87239-kube-api-access-tpbtk\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.812580 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.812554    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlbz\" (UniqueName: \"kubernetes.io/projected/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-kube-api-access-jxlbz\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:55.812838 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.812813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmkw\" (UniqueName: \"kubernetes.io/projected/8cdac693-b083-4a91-bd5c-ea56d23bd771-kube-api-access-qzmkw\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.813826 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.813805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdbx\" (UniqueName: \"kubernetes.io/projected/42f1ee76-96ba-4983-b33a-b7321b678045-kube-api-access-6bdbx\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.813945 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.813924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wql\" (UniqueName: \"kubernetes.io/projected/17928b13-2c16-48bd-9cad-e8920d246891-kube-api-access-j2wql\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.814091 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.814074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgxb\" (UniqueName: \"kubernetes.io/projected/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-kube-api-access-6rgxb\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.892660 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.892584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:28:55.899092 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.899073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:28:55.903253 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.903207    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bce3daf_4de1_4fea_89e8_cd891e432355.slice/crio-7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb WatchSource:0}: Error finding container 7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb: Status 404 returned error can't find the container with id 7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb
Nov 03 20:28:55.905446 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.905425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:28:55.907274 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.907250    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f1ee76_96ba_4983_b33a_b7321b678045.slice/crio-e126d66b3b61aa82bb3e0bea2010631108c5a519474bb139f534d5f90eadfc5a WatchSource:0}: Error finding container e126d66b3b61aa82bb3e0bea2010631108c5a519474bb139f534d5f90eadfc5a: Status 404 returned error can't find the container with id e126d66b3b61aa82bb3e0bea2010631108c5a519474bb139f534d5f90eadfc5a
Nov 03 20:28:55.913662 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.913646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:28:55.914100 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.914084    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be5ee45_4c57_4f1f_a3a1_075c2dca3c90.slice/crio-cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14 WatchSource:0}: Error finding container cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14: Status 404 returned error can't find the container with id cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14
Nov 03 20:28:55.919543 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.919525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:28:55.922287 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.922265    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7871e65_e639_4c12_ad40_4a2dd1b90687.slice/crio-43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0 WatchSource:0}: Error finding container 43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0: Status 404 returned error can't find the container with id 43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0
Nov 03 20:28:55.924693 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.924675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knqnr"
Nov 03 20:28:55.927662 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.927642    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda641934f_d459_46af_be40_6aee398f3b82.slice/crio-ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1 WatchSource:0}: Error finding container ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1: Status 404 returned error can't find the container with id ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1
Nov 03 20:28:55.933451 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.933426    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c38cada_a1d9_40a5_83c2_ac7157fb9975.slice/crio-59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59 WatchSource:0}: Error finding container 59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59: Status 404 returned error can't find the container with id 59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59
Nov 03 20:28:55.950996 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.950977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:28:55.959765 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.959747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:28:55.959954 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.959936    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19efb517_1936_48f3_ab42_bd7ee7a87239.slice/crio-d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532 WatchSource:0}: Error finding container d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532: Status 404 returned error can't find the container with id d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532
Nov 03 20:28:55.967887 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.967864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:28:55.968141 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.968123    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdac693_b083_4a91_bd5c_ea56d23bd771.slice/crio-a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c WatchSource:0}: Error finding container a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c: Status 404 returned error can't find the container with id a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c
Nov 03 20:28:55.977372 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:28:55.977338    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17928b13_2c16_48bd_9cad_e8920d246891.slice/crio-29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b WatchSource:0}: Error finding container 29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b: Status 404 returned error can't find the container with id 29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b
Nov 03 20:28:55.984038 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:55.984019    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:28:56.252325 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.252228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:56.252325 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.252312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:56.252566 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.252398    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:56.252566 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.252441    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:56.252566 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.252464    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.252449551 +0000 UTC m=+4.235671015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:56.252566 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.252496    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.252479394 +0000 UTC m=+4.235700873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:56.353667 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.353613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:56.353825 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.353765    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:56.353825 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.353783    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:56.353825 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.353793    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:56.353924 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:56.353849    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:28:57.353832711 +0000 UTC m=+4.337054177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:56.587569 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.587533    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerStarted","Data":"29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b"}
Nov 03 20:28:56.589229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.589204    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knqnr" event={"ID":"2c38cada-a1d9-40a5-83c2-ac7157fb9975","Type":"ContainerStarted","Data":"59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59"}
Nov 03 20:28:56.590260 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.590238    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kxpt4" event={"ID":"a641934f-d459-46af-be40-6aee398f3b82","Type":"ContainerStarted","Data":"ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1"}
Nov 03 20:28:56.591359 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.591321    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7dw2d" event={"ID":"c7871e65-e639-4c12-ad40-4a2dd1b90687","Type":"ContainerStarted","Data":"43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0"}
Nov 03 20:28:56.592601 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.592575    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vmcgc" event={"ID":"42f1ee76-96ba-4983-b33a-b7321b678045","Type":"ContainerStarted","Data":"e126d66b3b61aa82bb3e0bea2010631108c5a519474bb139f534d5f90eadfc5a"}
Nov 03 20:28:56.593903 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.593878    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9k7ks" event={"ID":"4bce3daf-4de1-4fea-89e8-cd891e432355","Type":"ContainerStarted","Data":"7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb"}
Nov 03 20:28:56.595116 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.595095    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c"}
Nov 03 20:28:56.596164 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.596144    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf" event={"ID":"19efb517-1936-48f3-ab42-bd7ee7a87239","Type":"ContainerStarted","Data":"d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532"}
Nov 03 20:28:56.597205 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.597186    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hk7dd" event={"ID":"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90","Type":"ContainerStarted","Data":"cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14"}
Nov 03 20:28:56.758020 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.757949    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:56 +0000 UTC" deadline="2028-08-08 08:12:19.171166992 +0000 UTC"
Nov 03 20:28:56.758020 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:56.758007    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15443h43m22.413163053s"
Nov 03 20:28:57.262832 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.262794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:57.263037 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.262879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:57.263037 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.263010    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:57.263149 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.263072    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:59.263053062 +0000 UTC m=+6.246274533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:57.263149 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.263141    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:57.263264 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.263177    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:28:59.26316408 +0000 UTC m=+6.246385551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:57.363641 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.363598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:57.363899 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.363860    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:57.363899 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.363884    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:57.363899 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.363897    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:57.364139 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.363981    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:28:59.363962312 +0000 UTC m=+6.347183793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:57.580611 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.580532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:57.580782 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.580541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:57.580782 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.580689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:57.580782 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.580684    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:28:57.580939 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.580796    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:28:57.580939 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:57.580881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:28:57.759246 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.759156    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:56 +0000 UTC" deadline="2028-06-09 10:24:11.079451209 +0000 UTC"
Nov 03 20:28:57.759246 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:57.759199    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14005h55m13.320257665s"
Nov 03 20:28:59.281639 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:59.281595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:59.282263 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:59.281763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:59.282263 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.281795    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:59.282263 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.281873    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:59.282263 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.281878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:29:03.281857447 +0000 UTC m=+10.265078927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:28:59.282263 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.281949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:03.281932924 +0000 UTC m=+10.265154393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:28:59.384106 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:59.384062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:59.384321 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.384246    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:28:59.384321 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.384268    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:28:59.384321 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.384281    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:59.384518 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.384338    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:29:03.384322136 +0000 UTC m=+10.367543609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:28:59.583605 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:59.583523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:28:59.583778 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.583685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:28:59.583778 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:59.583764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:28:59.583892 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.583842    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:28:59.583951 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:28:59.583893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:28:59.584002 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:28:59.583966    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:00.616721 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:00.616446    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba15cf602964f30d9df06d36baaae3a5" containerID="796e1aebe4685889ab7430fa65c3a69062e94c71ee6502294df09a8eb67a4705" exitCode=0
Nov 03 20:29:00.618527 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:00.616742    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ba15cf602964f30d9df06d36baaae3a5","Type":"ContainerDied","Data":"796e1aebe4685889ab7430fa65c3a69062e94c71ee6502294df09a8eb67a4705"}
Nov 03 20:29:01.580653 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.580607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:01.580653 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.580654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:01.580888 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.580622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:01.580888 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:01.580772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:01.580888 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:01.580849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:01.581052 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:01.580930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:01.622455 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.622023    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ba15cf602964f30d9df06d36baaae3a5","Type":"ContainerStarted","Data":"d821887e95a24f913bf9a83ba979e752664517672957fd30a19fa0bc19d14e2d"}
Nov 03 20:29:01.624706 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.624678    2578 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="2b6a1623b7102203df31717183b412182c565e5a08457494747e5d9c30f47688" exitCode=0
Nov 03 20:29:01.624820 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.624747    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerDied","Data":"2b6a1623b7102203df31717183b412182c565e5a08457494747e5d9c30f47688"}
Nov 03 20:29:01.628787 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.628303    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9k7ks" event={"ID":"4bce3daf-4de1-4fea-89e8-cd891e432355","Type":"ContainerStarted","Data":"00ffb8acd4c15d81234df43fbeab42096bffc303e6bb599d934c19a0913767d6"}
Nov 03 20:29:01.631666 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.631640    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ebc5d4efd8a63d505e3fa8dc27ffb1de","Type":"ContainerStarted","Data":"4b05b3620335ff600dcbaaba98c39ea5174b608163a35fbda19ee805218914c2"}
Nov 03 20:29:01.634589 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.634552    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf" event={"ID":"19efb517-1936-48f3-ab42-bd7ee7a87239","Type":"ContainerStarted","Data":"5b7cc938f0f8e0807812b7b6c70dd772ad46c596e1553670734526a34304ce74"}
Nov 03 20:29:01.748348 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.748296    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" podStartSLOduration=6.74827808 podStartE2EDuration="6.74827808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.74780684 +0000 UTC m=+8.731028329" watchObservedRunningTime="2026-11-03 20:29:01.74827808 +0000 UTC m=+8.731499568"
Nov 03 20:29:01.874889 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.874821    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9k7ks" podStartSLOduration=3.024497938 podStartE2EDuration="7.874802516s" totalImagesPullingTime="4.850304578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.873775769 +0000 UTC m=+8.856997258" watchObservedRunningTime="2026-11-03 20:29:01.874802516 +0000 UTC m=+8.858024007"
Nov 03 20:29:01.997731 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:01.997673    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal" podStartSLOduration=5.997658659 podStartE2EDuration="5.997658659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.99765578 +0000 UTC m=+8.980877266" watchObservedRunningTime="2026-11-03 20:29:01.997658659 +0000 UTC m=+8.980880147"
Nov 03 20:29:03.161873 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.161839    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:29:03.162662 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.162494    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:29:03.171980 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.171957    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:29:03.172633 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.172571    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:29:03.316897 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.316805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:03.316897 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.316873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:03.317121 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.317007    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:03.317121 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.317071    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:29:11.317050884 +0000 UTC m=+18.300272364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:03.317243 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.317143    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:03.317243 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.317181    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:11.317168969 +0000 UTC m=+18.300390450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:03.417741 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.417705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:03.417916 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.417901    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:29:03.417999 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.417926    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:29:03.417999 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.417939    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:03.418088 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.418000    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:29:11.417980351 +0000 UTC m=+18.401201834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:03.570680 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.570657    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:29:03.580783 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.580751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:03.580890 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.580880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:03.580951 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.580895    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:03.580951 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.580948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:03.581066 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.580999    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:03.581066 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:03.580946    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:03.640422 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.640383    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf" event={"ID":"19efb517-1936-48f3-ab42-bd7ee7a87239","Type":"ContainerStarted","Data":"58c9ff0f1e9c1fabb8671ac37561398134ed54b022547de5524ab51cc518ac01"}
Nov 03 20:29:03.641979 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.641938    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kxpt4" event={"ID":"a641934f-d459-46af-be40-6aee398f3b82","Type":"ContainerStarted","Data":"de481230c641611e09a0ef72012cbb5e10743f003db1a774f95afc752592cac7"}
Nov 03 20:29:03.765757 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:03.765707    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kxpt4" podStartSLOduration=2.692884438 podStartE2EDuration="9.765692786s" totalImagesPullingTime="7.072808348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:03.765419202 +0000 UTC m=+10.748640692" watchObservedRunningTime="2026-11-03 20:29:03.765692786 +0000 UTC m=+10.748914276"
Nov 03 20:29:04.486478 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:04.486372    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:29:03.570675006Z","UUID":"e3f46ed4-9e80-45b0-b4cb-c5b3c8ef8322","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:29:04.487783 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:04.487755    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:29:04.487783 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:04.487782    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:29:05.580791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:05.580060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:05.580791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:05.580106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:05.580791 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:05.580204    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:05.580791 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:05.580617    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:05.580791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:05.580679    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:05.580791 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:05.580751    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:07.580060 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:07.579976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:07.580060 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:07.580024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:07.580560 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:07.579983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:07.580560 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:07.580134    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:07.580560 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:07.580201    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:07.580560 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:07.580274    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:07.654915 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:07.654879    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knqnr" event={"ID":"2c38cada-a1d9-40a5-83c2-ac7157fb9975","Type":"ContainerStarted","Data":"49e113442c79f5f227f5f490232d4d030dad59e0019e61451cd1bddc9ae312a5"}
Nov 03 20:29:07.808698 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:07.808658    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-knqnr" podStartSLOduration=2.429568485 podStartE2EDuration="13.808645212s" totalImagesPullingTime="11.379076727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:07.808311532 +0000 UTC m=+14.791533018" watchObservedRunningTime="2026-11-03 20:29:07.808645212 +0000 UTC m=+14.791866697"
Nov 03 20:29:08.660105 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:08.659860    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf" event={"ID":"19efb517-1936-48f3-ab42-bd7ee7a87239","Type":"ContainerStarted","Data":"f80d9f9b71d6753c6740578b85ae2b51873c58f00adee9d6b2ef47c467de2ea9"}
Nov 03 20:29:08.661900 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:08.661809    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hk7dd" event={"ID":"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90","Type":"ContainerStarted","Data":"7d567ffe4f92b4e64a2434c732f97e17b3187c0f0e98b63e86ed99743942fe39"}
Nov 03 20:29:08.779841 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:08.779783    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf" podStartSLOduration=3.269068289 podStartE2EDuration="14.779764421s" totalImagesPullingTime="11.510696132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:08.779475973 +0000 UTC m=+15.762697459" watchObservedRunningTime="2026-11-03 20:29:08.779764421 +0000 UTC m=+15.762985907"
Nov 03 20:29:08.895640 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:08.895587    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hk7dd" podStartSLOduration=2.572656189 podStartE2EDuration="14.895573456s" totalImagesPullingTime="12.322917267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:08.895438657 +0000 UTC m=+15.878660144" watchObservedRunningTime="2026-11-03 20:29:08.895573456 +0000 UTC m=+15.878794941"
Nov 03 20:29:09.580710 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:09.580676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:09.580836 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:09.580732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:09.580836 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:09.580745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:09.580913 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:09.580834    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:09.580913 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:09.580877    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:09.581003 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:09.580974    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:09.664724 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:09.664696    2578 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="a5a13023c3331416ea76819bc0d3970d40fe4fe35c992885cd75639c8ddf6d0f" exitCode=0
Nov 03 20:29:09.665203 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:09.664782    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerDied","Data":"a5a13023c3331416ea76819bc0d3970d40fe4fe35c992885cd75639c8ddf6d0f"}
Nov 03 20:29:10.668234 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:10.668192    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7dw2d" event={"ID":"c7871e65-e639-4c12-ad40-4a2dd1b90687","Type":"ContainerStarted","Data":"5c63a00a3afde228ce84ef0ec28de4814efd4be2737fdd28cfa102454b328297"}
Nov 03 20:29:10.669599 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:10.669575    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vmcgc" event={"ID":"42f1ee76-96ba-4983-b33a-b7321b678045","Type":"ContainerStarted","Data":"c3f1562badc10cf841a860d693ed9c4d41c25b28ae71a51cf433b60c37b93180"}
Nov 03 20:29:10.819126 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:10.819076    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7dw2d" podStartSLOduration=4.515384761 podStartE2EDuration="16.81906361s" totalImagesPullingTime="12.303678849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:10.81876534 +0000 UTC m=+17.801986825" watchObservedRunningTime="2026-11-03 20:29:10.81906361 +0000 UTC m=+17.802285156"
Nov 03 20:29:10.935948 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:10.935866    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vmcgc" podStartSLOduration=2.9277076429999997 podStartE2EDuration="16.935853076s" totalImagesPullingTime="14.008145433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:10.93561133 +0000 UTC m=+17.918832814" watchObservedRunningTime="2026-11-03 20:29:10.935853076 +0000 UTC m=+17.919074595"
Nov 03 20:29:11.387665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.387627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:11.387907 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.387686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:11.387907 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.387779    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:11.387907 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.387794    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:11.387907 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.387842    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.387826623 +0000 UTC m=+34.371048098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:11.387907 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.387859    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.387851233 +0000 UTC m=+34.371072696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:11.488517 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.488489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:11.488664 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.488607    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:29:11.488664 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.488620    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:29:11.488664 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.488629    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:11.488759 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.488679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.488662635 +0000 UTC m=+34.471884114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:11.580305 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.580273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:11.580305 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.580290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:11.580544 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.580406    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:11.580544 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.580455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:11.580649 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.580544    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:11.580649 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:11.580605    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:11.693473 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.693441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.944959 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:11.944708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.144550 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:12.144516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.677416 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:12.677387    2578 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="4e4149f799c81b7d6743b3c63387f2dd14ee28b2ea4af729ddc39367ff6fa033" exitCode=0
Nov 03 20:29:12.677627 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:12.677430    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerDied","Data":"4e4149f799c81b7d6743b3c63387f2dd14ee28b2ea4af729ddc39367ff6fa033"}
Nov 03 20:29:13.580653 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:13.580616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:13.581123 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:13.580713    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:13.581123 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:13.580741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:13.581123 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:13.580836    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:13.581123 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:13.580864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:13.581123 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:13.580911    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:13.832770 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:13.832705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:14.031626 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:14.031596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:14.683870 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:14.683836    2578 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="563fc0290f216d3fcb93ea337fd75bf7737e39ab34d3b00952cb96a4b7591cb1" exitCode=0
Nov 03 20:29:14.684267 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:14.683887    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerDied","Data":"563fc0290f216d3fcb93ea337fd75bf7737e39ab34d3b00952cb96a4b7591cb1"}
Nov 03 20:29:15.580912 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:15.580878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:15.580912 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:15.580907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:15.581143 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:15.580915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:15.581143 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:15.581003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:15.581239 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:15.581135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:15.581239 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:15.581214    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:15.628807 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:15.628781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.836252 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:15.836173    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.431507 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.431476    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.580440 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.580415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:17.580440 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.580439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:17.580610 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.580415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:17.580610 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:17.580500    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:17.580671 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:17.580612    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:17.580723 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:17.580702    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:17.692964 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.692900    2578 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="f8f44f1e7e523de89a1644ee5bb04c60d41eb2ba6bd3c03038831878f845c0c2" exitCode=0
Nov 03 20:29:17.692964 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.692941    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerDied","Data":"f8f44f1e7e523de89a1644ee5bb04c60d41eb2ba6bd3c03038831878f845c0c2"}
Nov 03 20:29:17.694532 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:17.694512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:18.697071 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:18.697038    2578 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="acc8cc27df731dfa6a70c9794d196df165256ee20df553e158fbd0814f71cc2e" exitCode=0
Nov 03 20:29:18.697561 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:18.697085    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerDied","Data":"acc8cc27df731dfa6a70c9794d196df165256ee20df553e158fbd0814f71cc2e"}
Nov 03 20:29:18.723797 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:18.723775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:18.947836 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:18.947769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:19.154058 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.154023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:19.567069 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.567041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:19.580149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.580124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:19.580287 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.580156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:19.580287 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:19.580228    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:19.580287 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.580253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:19.580494 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:19.580327    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:19.580494 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:19.580423    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:19.701600 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.701573    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zpntw" event={"ID":"17928b13-2c16-48bd-9cad-e8920d246891","Type":"ContainerStarted","Data":"70aa30e6a1fcdac01591cc2b6861fa55abc1c550330220fe46b84e258a4c4fe5"}
Nov 03 20:29:19.869929 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:19.869890    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zpntw" podStartSLOduration=8.705446107 podStartE2EDuration="25.869878358s" totalImagesPullingTime="17.164432251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:19.869372612 +0000 UTC m=+26.852594097" watchObservedRunningTime="2026-11-03 20:29:19.869878358 +0000 UTC m=+26.853099843"
Nov 03 20:29:20.245352 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:20.245275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:20.443719 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:20.443689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:21.287288 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.287254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:21.479613 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.479578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:21.580039 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.579976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:21.580155 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.579977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:21.580155 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:21.580087    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:21.580155 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.579984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:21.580305 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:21.580150    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:21.580305 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:21.580206    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:21.689151 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.689123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:21.887634 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:21.887611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:22.110916 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:22.110857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:22.305253 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:22.305227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:22.503753 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:22.503673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:22.714935 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:22.714907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.148747 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.148717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.349785 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.349760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.548473 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.548403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.580503 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.580483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:23.580628 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:23.580607    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:23.580691 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.580650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:23.580791 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:23.580772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:23.580851 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.580806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:23.580900 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:23.580888    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:23.753172 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:23.753131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.240570 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:24.240531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.440807 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:24.440782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.637450 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:24.637414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:25.140592 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:25.140558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:25.579957 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:25.579881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:25.579957 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:25.579927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:25.580370 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:25.579881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:25.580370 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:25.580001    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:25.580370 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:25.580053    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:25.580370 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:25.580119    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:25.731329 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:25.731302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:25.923757 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:25.923728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:26.123311 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:26.123282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:26.237177 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:26.237113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hk7dd_0be5ee45-4c57-4f1f-a3a1-075c2dca3c90/dns-node-resolver/0.log"
Nov 03 20:29:26.331117 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:26.331093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:26.529404 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:26.529329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:27.127126 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.127098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:27.237147 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.237122    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kxpt4_a641934f-d459-46af-be40-6aee398f3b82/node-ca/0.log"
Nov 03 20:29:27.405057 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.404988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:27.405057 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.405032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:27.405254 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.405137    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:27.405254 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.405138    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:27.405254 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.405184    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs podName:9285d5c2-ae97-46ad-974a-2e2cbce9b36e nodeName:}" failed. No retries permitted until 2026-11-03 20:29:59.405170647 +0000 UTC m=+66.388392123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs") pod "network-metrics-daemon-ms89f" (UID: "9285d5c2-ae97-46ad-974a-2e2cbce9b36e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:29:27.405254 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.405196    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret podName:7574d083-15a3-4cf9-9547-93c53eb8222a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:59.40519079 +0000 UTC m=+66.388412253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret") pod "global-pull-secret-syncer-nl7nq" (UID: "7574d083-15a3-4cf9-9547-93c53eb8222a") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:29:27.505485 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.505455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:27.505621 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.505604    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:29:27.505670 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.505627    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:29:27.505670 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.505637    2578 projected.go:196] Error preparing data for projected volume kube-api-access-s8cph for pod openshift-network-diagnostics/network-check-target-d82lk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:27.505742 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.505679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph podName:55f1e1e3-49b2-4665-b2a7-5e0e807d0cee nodeName:}" failed. No retries permitted until 2026-11-03 20:29:59.505665921 +0000 UTC m=+66.488887388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8cph" (UniqueName: "kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph") pod "network-check-target-d82lk" (UID: "55f1e1e3-49b2-4665-b2a7-5e0e807d0cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:29:27.580750 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.580726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:27.580866 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.580753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:27.580940 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.580914    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:27.580986 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.580958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:27.581048 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.581030    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:27.581136 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:27.581115    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:27.724791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.724738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:27.928670 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:27.928647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:28.128180 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:28.128152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:28.529694 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:28.529622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.325941 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:29.325910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.580229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:29.580151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:29.580229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:29.580178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:29.580516 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:29.580156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:29.580516 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:29.580254    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:29.580516 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:29.580347    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:29.580516 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:29.580420    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:29.726088 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:29.726059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.923492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:29.923461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.128680 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:30.128651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.328893 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:30.328824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.530331 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:30.530298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.730232 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:30.730188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.924540 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:30.924507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.123367 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.123320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.323112 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.323075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.521064 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.520986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.580674 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.580641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:31.580835 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.580646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:31.580835 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:31.580775    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:31.580927 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:31.580872    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:31.580927 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.580653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:31.581008 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:31.580967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:31.723214 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.723165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.931293 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:31.931254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:32.257171 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:32.257095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:32.470066 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:32.470011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:32.731651 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:32.731607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:32.929699 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:32.929659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:33.137358 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:33.137295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:33.404638 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:33.404557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:33.581409 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:33.581377    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:33.581409 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:33.581413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:33.581641 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:33.581389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:33.581641 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:33.581490    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:33.581641 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:33.581597    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:33.581799 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:33.581685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:34.732855 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.732728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:29:34.733437 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.733177    2578 generic.go:356] "Generic (PLEG): container finished" podID="8cdac693-b083-4a91-bd5c-ea56d23bd771" containerID="738ac27585078545917c948af9caba3367bf6704c07cf73ca05ef14033d2d355" exitCode=1
Nov 03 20:29:34.733437 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.733247    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"522d619cac6ad49c0bd6279d5502be3aed62976ee2c7adfefb70c201c564e129"}
Nov 03 20:29:34.733437 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.733358    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"fd6f4c226e95ca8a5e0369e35a5e7dabfaada7395c0fbd760a392c4bd4a14d8f"}
Nov 03 20:29:34.733437 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.733394    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerDied","Data":"738ac27585078545917c948af9caba3367bf6704c07cf73ca05ef14033d2d355"}
Nov 03 20:29:34.733437 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.733423    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"76dba1432a5677923b7192d34f9ee3e2b385e1b6f8abf68417876d13dbc2bc31"}
Nov 03 20:29:34.998449 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:34.998355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:35.195303 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.195275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:35.580322 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.580290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:35.580322 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.580307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:35.580524 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.580291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:35.580524 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:35.580429    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:35.580584 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:35.580542    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:35.580631 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:35.580614    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:35.737973 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.737951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:29:35.738374 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.738266    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"7812b94d6ee345754523610160e0ecaa484aee2c769c8968b814348d050c0a88"}
Nov 03 20:29:35.738374 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:35.738345    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"4ef71ef03c67c5bf8bda697acc13ebdb541f4a57a37b12686360fb39404e7912"}
Nov 03 20:29:37.579905 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:37.579867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:37.580408 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:37.579867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:37.580408 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:37.580009    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:37.580408 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:37.579874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:37.580408 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:37.580054    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:37.580408 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:37.580122    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:37.745672 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:37.745642    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:29:37.745980 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:37.745961    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"e67af6cfb0af9dd38400890e8e7cb0ee29365ce9f1e68bcd56aabd2c2f4b63df"}
Nov 03 20:29:38.598447 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:38.598414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:38.800687 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:38.800656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:39.580296 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.580099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:39.580539 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.580175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:39.580539 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:39.580411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:39.580539 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.580183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:39.580539 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:39.580462    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:39.580539 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:39.580523    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:39.752608 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.752583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:29:39.753053 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.752883    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"f7fc65d692c842fc4ec5f34c330ff783f4333a9a2aab659f698fadc5f744e87b"}
Nov 03 20:29:39.753174 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.753156    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:29:39.753244 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.753233    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:29:39.753299 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.753288    2578 scope.go:122] "RemoveContainer" containerID="738ac27585078545917c948af9caba3367bf6704c07cf73ca05ef14033d2d355"
Nov 03 20:29:39.769188 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:39.769167    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:29:40.761560 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:40.761533    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:29:40.761943 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:40.761894    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" event={"ID":"8cdac693-b083-4a91-bd5c-ea56d23bd771","Type":"ContainerStarted","Data":"1e887a1fa79c7d8d82e3443ee8c1b2b2f6b2ffeeed23df41494a04aacf8ac853"}
Nov 03 20:29:40.762074 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:40.762062    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:29:40.776603 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:40.776578    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:29:40.941549 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:40.941496    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs" podStartSLOduration=8.545854117 podStartE2EDuration="46.941483995s" totalImagesPullingTime="38.395629878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:40.941023081 +0000 UTC m=+47.924244566" watchObservedRunningTime="2026-11-03 20:29:40.941483995 +0000 UTC m=+47.924705480"
Nov 03 20:29:41.580535 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:41.580497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:41.580535 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:41.580524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:41.580785 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:41.580646    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:41.580785 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:41.580691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:41.580785 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:41.580761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:41.580942 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:41.580815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:43.195468 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:43.195435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:43.395243 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:43.395216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:43.580987 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:43.580904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:43.581142 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:43.581028    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:43.581142 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:43.581072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:43.581142 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:43.581135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:43.581299 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:43.581209    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:43.581299 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:43.581232    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:44.630002 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:44.629284    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nl7nq"]
Nov 03 20:29:44.630478 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:44.630096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:44.630478 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:44.630221    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:44.631609 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:44.631588    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d82lk"]
Nov 03 20:29:44.631844 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:44.631824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:44.631957 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:44.631937    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:44.632471 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:44.632449    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ms89f"]
Nov 03 20:29:44.632750 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:44.632734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:44.632960 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:44.632939    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:46.062476 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:46.062409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:46.276179 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:46.276146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:46.580828 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:46.580792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:46.580828 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:46.580814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:46.580828 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:46.580794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:46.581059 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:46.580898    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:46.581059 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:46.580957    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:46.581059 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:46.581043    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:47.431296 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:47.431266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.632908 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:47.632863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:48.142721 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:48.142686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:48.339949 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:48.339924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:48.554952 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:48.554882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:48.580809 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:48.580787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:48.580809 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:48.580802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:48.580943 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:48.580858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:48.580943 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:48.580930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nl7nq" podUID="7574d083-15a3-4cf9-9547-93c53eb8222a"
Nov 03 20:29:48.581053 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:48.581031    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d82lk" podUID="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee"
Nov 03 20:29:48.581112 ip-10-0-135-141 kubenswrapper[2578]: E1103 20:29:48.581100    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms89f" podUID="9285d5c2-ae97-46ad-974a-2e2cbce9b36e"
Nov 03 20:29:49.224069 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.223992    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.276094 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.276067    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeReady"
Nov 03 20:29:49.276217 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.276162    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:29:49.427032 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.427004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.442774 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.442746    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5vdpt"]
Nov 03 20:29:49.445697 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.444887    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j6bcz"]
Nov 03 20:29:49.445697 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.445260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.446802 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.446783    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vt4cx"]
Nov 03 20:29:49.446926 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.446905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.449254 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.449229    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6bcz"]
Nov 03 20:29:49.449369 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.449267    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5vdpt"]
Nov 03 20:29:49.449369 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.449362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.460837 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.460819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12201904-e96b-4326-b87d-b5ac7539e37e-cert\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.460909 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.460857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.460960 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.460945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmww\" (UniqueName: \"kubernetes.io/projected/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-api-access-mcmww\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.460995 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.460978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-data-volume\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.461029 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0305c1b2-ad03-4e5b-847f-64b2828fedc2-tmp-dir\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.461060 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4r2z\" (UniqueName: \"kubernetes.io/projected/0305c1b2-ad03-4e5b-847f-64b2828fedc2-kube-api-access-c4r2z\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.461099 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.461135 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfnj\" (UniqueName: \"kubernetes.io/projected/12201904-e96b-4326-b87d-b5ac7539e37e-kube-api-access-lxfnj\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.461167 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0305c1b2-ad03-4e5b-847f-64b2828fedc2-config-volume\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.461245 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-crio-socket\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.461303 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.461286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0305c1b2-ad03-4e5b-847f-64b2828fedc2-metrics-tls\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.469411 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.469387    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vt4cx"]
Nov 03 20:29:49.498468 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.498423    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:29:49.498577 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.498556    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:29:49.498629 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.498568    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9nsh4\""
Nov 03 20:29:49.499452 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.499438    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:29:49.499501 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.499460    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdt7f\""
Nov 03 20:29:49.550181 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550158    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:29:49.550284 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550203    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:29:49.550284 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550223    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:29:49.550404 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550290    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9t9mk\""
Nov 03 20:29:49.550404 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550300    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:29:49.550404 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550305    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:29:49.550404 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.550372    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:29:49.561625 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0305c1b2-ad03-4e5b-847f-64b2828fedc2-config-volume\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-crio-socket\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0305c1b2-ad03-4e5b-847f-64b2828fedc2-metrics-tls\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12201904-e96b-4326-b87d-b5ac7539e37e-cert\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmww\" (UniqueName: \"kubernetes.io/projected/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-api-access-mcmww\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-data-volume\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0305c1b2-ad03-4e5b-847f-64b2828fedc2-tmp-dir\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561791    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-crio-socket\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4r2z\" (UniqueName: \"kubernetes.io/projected/0305c1b2-ad03-4e5b-847f-64b2828fedc2-kube-api-access-c4r2z\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562095 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.561895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfnj\" (UniqueName: \"kubernetes.io/projected/12201904-e96b-4326-b87d-b5ac7539e37e-kube-api-access-lxfnj\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.562656 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.562155    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-data-volume\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562656 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.562360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0305c1b2-ad03-4e5b-847f-64b2828fedc2-config-volume\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.562656 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.562386    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.562656 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.562416    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0305c1b2-ad03-4e5b-847f-64b2828fedc2-tmp-dir\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.565660 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.565640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0305c1b2-ad03-4e5b-847f-64b2828fedc2-metrics-tls\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.565733 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.565640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12201904-e96b-4326-b87d-b5ac7539e37e-cert\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.565733 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.565640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.622950 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.622925    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmww\" (UniqueName: \"kubernetes.io/projected/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-api-access-mcmww\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:49.622950 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.622946    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4r2z\" (UniqueName: \"kubernetes.io/projected/0305c1b2-ad03-4e5b-847f-64b2828fedc2-kube-api-access-c4r2z\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.623123 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.623039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfnj\" (UniqueName: \"kubernetes.io/projected/12201904-e96b-4326-b87d-b5ac7539e37e-kube-api-access-lxfnj\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.759304 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.759232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:49.764224 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.764206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:29:49.772113 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:49.772091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:29:50.188484 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.188245    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6bcz"]
Nov 03 20:29:50.188835 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:29:50.188813    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12201904_e96b_4326_b87d_b5ac7539e37e.slice/crio-7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5 WatchSource:0}: Error finding container 7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5: Status 404 returned error can't find the container with id 7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5
Nov 03 20:29:50.191254 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.191235    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5vdpt"]
Nov 03 20:29:50.191994 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:29:50.191968    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48361014_a0f8_4050_8fdf_bf7d7b70ddcb.slice/crio-f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd WatchSource:0}: Error finding container f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd: Status 404 returned error can't find the container with id f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd
Nov 03 20:29:50.192069 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.192012    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vt4cx"]
Nov 03 20:29:50.192605 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:29:50.192584    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0305c1b2_ad03_4e5b_847f_64b2828fedc2.slice/crio-3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86 WatchSource:0}: Error finding container 3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86: Status 404 returned error can't find the container with id 3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86
Nov 03 20:29:50.580830 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.580763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:50.581314 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.580764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:50.581314 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.580779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:50.633794 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.633746    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:29:50.633920 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.633904    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:29:50.634227 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.634209    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:29:50.634287 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.634226    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\""
Nov 03 20:29:50.634287 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.634241    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:29:50.634287 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.634208    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\""
Nov 03 20:29:50.634287 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.634226    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\""
Nov 03 20:29:50.783390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.783363    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt4cx" event={"ID":"48361014-a0f8-4050-8fdf-bf7d7b70ddcb","Type":"ContainerStarted","Data":"c8cbb4b6be7d20abc390761cc4be154dcb47bbac8871c281927badddd5fe03f6"}
Nov 03 20:29:50.783519 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.783420    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt4cx" event={"ID":"48361014-a0f8-4050-8fdf-bf7d7b70ddcb","Type":"ContainerStarted","Data":"f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd"}
Nov 03 20:29:50.784493 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.784469    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6bcz" event={"ID":"12201904-e96b-4326-b87d-b5ac7539e37e","Type":"ContainerStarted","Data":"7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5"}
Nov 03 20:29:50.785424 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:50.785406    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5vdpt" event={"ID":"0305c1b2-ad03-4e5b-847f-64b2828fedc2","Type":"ContainerStarted","Data":"3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86"}
Nov 03 20:29:51.203108 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.203081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:51.418775 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.418747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:51.567434 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.567361    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb95cdfdd-vrl5j"]
Nov 03 20:29:51.569635 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.569612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.618246 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.618214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676604    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676707    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676779    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676870    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.676942    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.677045    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.677117    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g5f8g\""
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.677131    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb95cdfdd-vrl5j"]
Nov 03 20:29:51.677199 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.677174    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:29:51.777409 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.777409 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.777638 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.777638 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.777638 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.777638 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.777638 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.777600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.778620 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.778330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.778620 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.778567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.778768 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.778642    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.778831 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.778808    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.781975 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.781933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.782251 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.782199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.791655 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.791626    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt4cx" event={"ID":"48361014-a0f8-4050-8fdf-bf7d7b70ddcb","Type":"ContainerStarted","Data":"a970ef4977d3851b5799e6fddc55764fb911a86c0c4b0bceeff21a9e02782c8e"}
Nov 03 20:29:51.840135 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.840102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:51.883505 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:51.883470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:29:52.260975 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:29:52.260939    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336018cd_e7cc_4f23_809a_8b67aebd020d.slice/crio-9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2 WatchSource:0}: Error finding container 9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2: Status 404 returned error can't find the container with id 9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2
Nov 03 20:29:52.261372 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:52.261348    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb95cdfdd-vrl5j"]
Nov 03 20:29:52.796890 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:52.796845    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb95cdfdd-vrl5j" event={"ID":"336018cd-e7cc-4f23-809a-8b67aebd020d","Type":"ContainerStarted","Data":"9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2"}
Nov 03 20:29:53.513813 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.513775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.727714 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.727675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.803149 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.803116    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5vdpt" event={"ID":"0305c1b2-ad03-4e5b-847f-64b2828fedc2","Type":"ContainerStarted","Data":"4248ccd12cedb79a78e883b385da36c857a2f71a9e0a15162db151d2a045cf22"}
Nov 03 20:29:53.803570 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.803176    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5vdpt" event={"ID":"0305c1b2-ad03-4e5b-847f-64b2828fedc2","Type":"ContainerStarted","Data":"73aa745332fde3642d143ddeb2db3c6e934823190b44b92eec42d2dedabec2e2"}
Nov 03 20:29:53.803827 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.803808    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:29:53.907594 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.907557    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"]
Nov 03 20:29:53.910730 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.910705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:53.927857 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.927831    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"]
Nov 03 20:29:53.930609 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.930575    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.957402 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.957324    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5vdpt" podStartSLOduration=2.279985765 podStartE2EDuration="4.957307043s" totalImagesPullingTime="2.677321278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:53.949995637 +0000 UTC m=+60.933217119" watchObservedRunningTime="2026-11-03 20:29:53.957307043 +0000 UTC m=+60.940528531"
Nov 03 20:29:53.976031 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.975996    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:29:53.995492 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.995461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:53.995609 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.995501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchxh\" (UniqueName: \"kubernetes.io/projected/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-kube-api-access-hchxh\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:53.995609 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.995536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:53.995694 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:53.995614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.000204 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.000183    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:29:54.000204 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.000194    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 20:29:54.000399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.000213    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:29:54.000399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.000229    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 20:29:54.000399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.000286    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qfcmp\""
Nov 03 20:29:54.096917 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.096851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.097074 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.096943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.097074 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.096966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchxh\" (UniqueName: \"kubernetes.io/projected/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-kube-api-access-hchxh\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.097074 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.096986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.098435 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.098404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.099832 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.099802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.099949 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.099921    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.159514 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.159483    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchxh\" (UniqueName: \"kubernetes.io/projected/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-kube-api-access-hchxh\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.231396 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.231365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:29:54.601903 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:29:54.601872    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cae6ab2_e17e_4b19_8e99_f36f7072a3b7.slice/crio-15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda WatchSource:0}: Error finding container 15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda: Status 404 returned error can't find the container with id 15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda
Nov 03 20:29:54.602708 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.602686    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"]
Nov 03 20:29:54.807604 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.807575    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6bcz" event={"ID":"12201904-e96b-4326-b87d-b5ac7539e37e","Type":"ContainerStarted","Data":"8734be332442737cb50a883cd2cb778e041c72c1d2dd59e66419cd2dbcb51bfc"}
Nov 03 20:29:54.810285 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.810257    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt4cx" event={"ID":"48361014-a0f8-4050-8fdf-bf7d7b70ddcb","Type":"ContainerStarted","Data":"a6cd835bcce6612db685a3c2d20bac1aae5903e955b8660aab3e91be47971cc7"}
Nov 03 20:29:54.811570 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.811547    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k" event={"ID":"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7","Type":"ContainerStarted","Data":"15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda"}
Nov 03 20:29:54.923367 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:54.923248    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j6bcz" podStartSLOduration=2.334414034 podStartE2EDuration="5.923234013s" totalImagesPullingTime="3.588819979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:54.923011347 +0000 UTC m=+61.906232832" watchObservedRunningTime="2026-11-03 20:29:54.923234013 +0000 UTC m=+61.906455498"
Nov 03 20:29:55.042781 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:55.042720    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vt4cx" podStartSLOduration=2.411654434 podStartE2EDuration="6.042704839s" totalImagesPullingTime="3.631050405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:29:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.041944539 +0000 UTC m=+62.025166049" watchObservedRunningTime="2026-11-03 20:29:55.042704839 +0000 UTC m=+62.025926363"
Nov 03 20:29:55.817973 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:55.817934    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb95cdfdd-vrl5j" event={"ID":"336018cd-e7cc-4f23-809a-8b67aebd020d","Type":"ContainerStarted","Data":"b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421"}
Nov 03 20:29:55.936311 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:55.936193    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cb95cdfdd-vrl5j" podStartSLOduration=1.542436245 podStartE2EDuration="4.936175891s" totalImagesPullingTime="3.393739646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.935331069 +0000 UTC m=+62.918552554" watchObservedRunningTime="2026-11-03 20:29:55.936175891 +0000 UTC m=+62.919397379"
Nov 03 20:29:56.233222 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:56.233145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:56.433390 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:56.433328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:56.629875 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:56.629836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:56.823660 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:56.823577    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k" event={"ID":"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7","Type":"ContainerStarted","Data":"8803710681f38e8cd1e0479ef04a7e35ecdaa0f22b75cc6622bd0da0694718d1"}
Nov 03 20:29:56.823660 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:56.823653    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k" event={"ID":"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7","Type":"ContainerStarted","Data":"9076899de52fb641f920be64b394e4561507f93fd5b74d396e392d63134548ae"}
Nov 03 20:29:56.941480 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:56.941427    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k" podStartSLOduration=2.084452458 podStartE2EDuration="3.941414494s" totalImagesPullingTime="1.856962036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:56.940744124 +0000 UTC m=+63.923965609" watchObservedRunningTime="2026-11-03 20:29:56.941414494 +0000 UTC m=+63.924636020"
Nov 03 20:29:59.287977 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.287944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.445134 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.445094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:59.445306 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.445148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:59.546438 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.546353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:59.547907 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.547890    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:29:59.547962 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.547899    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:29:59.558326 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.558303    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:59.558463 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.558444    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:59.598039 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.598018    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:29:59.644864 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.644833    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\""
Nov 03 20:29:59.648802 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.648784    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\""
Nov 03 20:29:59.654009 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.653992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:29:59.657584 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.657569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:29:59.658964 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.658943    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:29:59.670726 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.670705    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:29:59.872804 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.872772    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.971479 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.971453    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\""
Nov 03 20:29:59.972501 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:29:59.972483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:30:00.074609 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:00.074557    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7574d083_15a3_4cf9_9547_93c53eb8222a.slice/crio-2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915 WatchSource:0}: Error finding container 2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915: Status 404 returned error can't find the container with id 2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915
Nov 03 20:30:00.075708 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.075687    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nl7nq"]
Nov 03 20:30:00.121203 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.121179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:00.122158 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:00.122132    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9285d5c2_ae97_46ad_974a_2e2cbce9b36e.slice/crio-e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb WatchSource:0}: Error finding container e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb: Status 404 returned error can't find the container with id e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb
Nov 03 20:30:00.124309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.124289    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ms89f"]
Nov 03 20:30:00.333169 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:00.333141    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f1e1e3_49b2_4665_b2a7_5e0e807d0cee.slice/crio-3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e WatchSource:0}: Error finding container 3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e: Status 404 returned error can't find the container with id 3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e
Nov 03 20:30:00.333633 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.333614    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d82lk"]
Nov 03 20:30:00.382441 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.382385    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:00.524185 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.524152    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"]
Nov 03 20:30:00.528134 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.528105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.553649 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.553625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.553765 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.553656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrv6d\" (UniqueName: \"kubernetes.io/projected/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-kube-api-access-rrv6d\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.553765 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.553676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.553765 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.553698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.586098 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.586067    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 20:30:00.586098 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.586084    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:30:00.586275 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.586117    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tmzf6\""
Nov 03 20:30:00.587597 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.587572    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"]
Nov 03 20:30:00.587829 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.587807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:00.654483 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.654450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.654611 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.654521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.654611 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.654552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrv6d\" (UniqueName: \"kubernetes.io/projected/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-kube-api-access-rrv6d\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.654611 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.654589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.655446 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.655407    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.656970 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.656951    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.657040 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.656969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.674397 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.674370    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4ksx"]
Nov 03 20:30:00.676882 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.676866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.718352 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.718320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrv6d\" (UniqueName: \"kubernetes.io/projected/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-kube-api-access-rrv6d\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.734948 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.734924    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:30:00.755182 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-textfile\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755282 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rd6\" (UniqueName: \"kubernetes.io/projected/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-kube-api-access-48rd6\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755282 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755379 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-tls\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755379 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-wtmp\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755379 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755504 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-root\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755504 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-metrics-client-ca\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.755571 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.755551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-sys\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.801674 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.801646    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:30:00.801802 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.801732    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4nsd9\""
Nov 03 20:30:00.802207 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.802189    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:30:00.828571 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.828543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:00.835625 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.835601    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nl7nq" event={"ID":"7574d083-15a3-4cf9-9547-93c53eb8222a","Type":"ContainerStarted","Data":"2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915"}
Nov 03 20:30:00.836696 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.836678    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d82lk" event={"ID":"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee","Type":"ContainerStarted","Data":"3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e"}
Nov 03 20:30:00.837753 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.837735    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms89f" event={"ID":"9285d5c2-ae97-46ad-974a-2e2cbce9b36e","Type":"ContainerStarted","Data":"e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb"}
Nov 03 20:30:00.840968 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.840953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:00.856081 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-tls\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856167 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-wtmp\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856167 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856167 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-root\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856423 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856184    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-root\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856423 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-metrics-client-ca\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856423 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-wtmp\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856423 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-sys\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856723 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-sys\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856723 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-textfile\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856723 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rd6\" (UniqueName: \"kubernetes.io/projected/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-kube-api-access-48rd6\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856723 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.856851 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.856723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-metrics-client-ca\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.857026 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.857005    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-textfile\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.857109 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.857087    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.858281 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.858257    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-tls\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.858898 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.858879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.918604 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.918551    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rd6\" (UniqueName: \"kubernetes.io/projected/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-kube-api-access-48rd6\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.990165 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:00.990138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:00.999602 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:00.999579    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af07fe5_98e6_4f65_a70f_1b5cfe9c1735.slice/crio-9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b WatchSource:0}: Error finding container 9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b: Status 404 returned error can't find the container with id 9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b
Nov 03 20:30:01.069041 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.068990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:01.207396 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:01.207364    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298b5083_4edb_4714_96e4_a9d9a9a4a1e1.slice/crio-157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03 WatchSource:0}: Error finding container 157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03: Status 404 returned error can't find the container with id 157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03
Nov 03 20:30:01.208036 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.208008    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"]
Nov 03 20:30:01.266883 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.266857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:01.844249 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.844212    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m" event={"ID":"298b5083-4edb-4714-96e4-a9d9a9a4a1e1","Type":"ContainerStarted","Data":"bc52e62a3ce58255dd2a17df1d287f376dea1c1087a2ea38a49010c5d4e8cbe5"}
Nov 03 20:30:01.844718 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.844289    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m" event={"ID":"298b5083-4edb-4714-96e4-a9d9a9a4a1e1","Type":"ContainerStarted","Data":"94d3919fb47b10f776720b6934f9554e75e1d4718401f5fcebfd8a774b74dd81"}
Nov 03 20:30:01.844718 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.844318    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m" event={"ID":"298b5083-4edb-4714-96e4-a9d9a9a4a1e1","Type":"ContainerStarted","Data":"157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03"}
Nov 03 20:30:01.846001 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.845974    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4ksx" event={"ID":"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735","Type":"ContainerStarted","Data":"9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b"}
Nov 03 20:30:01.884410 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.884374    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:01.884527 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.884457    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:01.891251 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.891222    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:01.976354 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:01.975743    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.392487 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.392453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.599094 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.599037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.855791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.855074    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms89f" event={"ID":"9285d5c2-ae97-46ad-974a-2e2cbce9b36e","Type":"ContainerStarted","Data":"dbd507e35a8d9a7e364c00aa96d43a265349459ca410539d651450fb61fcfce9"}
Nov 03 20:30:02.855791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.855143    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms89f" event={"ID":"9285d5c2-ae97-46ad-974a-2e2cbce9b36e","Type":"ContainerStarted","Data":"e72ddf524940e75b6ed46d5d9b17b00f2682d9fabaf7365fc70c7ea51e5b6c95"}
Nov 03 20:30:02.859738 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.859708    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4ksx" event={"ID":"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735","Type":"ContainerStarted","Data":"63bdaaef1c37182a92650028e2e86c1a245cb7502ef540180826edf132b82ee8"}
Nov 03 20:30:02.866785 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.865165    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:02.872129 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.872106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.880060 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.880039    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:02.880189 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.880145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.933395 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933330    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:30:02.933605 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933579    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:30:02.933605 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933614    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:30:02.933772 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933737    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:30:02.933772 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933768    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hl5hz\""
Nov 03 20:30:02.933871 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933779    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:30:02.933871 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.933837    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:30:02.960524 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.960506    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:30:02.960628 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.960550    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:30:02.972981 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.972953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973094 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.972993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973094 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973206 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973265 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973681 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973824 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973887 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.973956 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.973934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.974092 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.974072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.974140 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.974106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.974229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.974196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:02.974391 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:02.974375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.012272 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.012246    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:30:03.036440 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.036409    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:03.037326 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.037275    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ms89f" podStartSLOduration=67.295542952 podStartE2EDuration="1m9.037258624s" totalImagesPullingTime="1.741715672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.034853365 +0000 UTC m=+70.018074852" watchObservedRunningTime="2026-11-03 20:30:03.037258624 +0000 UTC m=+70.020480111"
Nov 03 20:30:03.075183 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075310 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.075665 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.075451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.076099 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.076081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.077256 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.076955    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.079687 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.079356    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.079687 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.079629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.079687 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.079655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.079891 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.079741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.080455 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.080411    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.080775 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.080573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.080775 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.080742    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.081246 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.081205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.082869 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.082846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.082961 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.082893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.138840 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.138809    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.193592 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.193534    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"]
Nov 03 20:30:03.211543 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.211517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.257730 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.256955    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"]
Nov 03 20:30:03.277275 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/056b04d8-72b3-4019-bddd-56a4469906de-tmp\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277360 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh977\" (UniqueName: \"kubernetes.io/projected/056b04d8-72b3-4019-bddd-56a4469906de-kube-api-access-gh977\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277444 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277522 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-grpc-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277632 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277684 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277730 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277788 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056b04d8-72b3-4019-bddd-56a4469906de-metrics-client-ca\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.277788 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.277756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.278768 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.278674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:03.301736 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.301716    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 20:30:03.301831 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.301786    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 20:30:03.302031 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.302009    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nczmv\""
Nov 03 20:30:03.302100 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.302087    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-en5e1gdr1rmv0\""
Nov 03 20:30:03.302156 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.302145    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 20:30:03.302225 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.302213    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 20:30:03.302277 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.302241    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 20:30:03.378514 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh977\" (UniqueName: \"kubernetes.io/projected/056b04d8-72b3-4019-bddd-56a4469906de-kube-api-access-gh977\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.378514 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.378514 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-grpc-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.378514 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.380309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.380309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.380309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.378967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056b04d8-72b3-4019-bddd-56a4469906de-metrics-client-ca\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.380309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.379002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.380309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.379055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/056b04d8-72b3-4019-bddd-56a4469906de-tmp\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.380309 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.380233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056b04d8-72b3-4019-bddd-56a4469906de-metrics-client-ca\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.381308 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.381264    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/056b04d8-72b3-4019-bddd-56a4469906de-tmp\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.388675 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.382589    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-grpc-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.388675 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.383147    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.389980 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.389953    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.390768 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.390460    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.391702 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.391668    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.392496 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.392450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.441467 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.441434    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh977\" (UniqueName: \"kubernetes.io/projected/056b04d8-72b3-4019-bddd-56a4469906de-kube-api-access-gh977\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.566824 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.566744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:03.611330 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.611294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:03.735688 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:03.735660    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9459b28f_1cb6_4b5f_b0df_f30ccd9f5470.slice/crio-b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68 WatchSource:0}: Error finding container b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68: Status 404 returned error can't find the container with id b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68
Nov 03 20:30:03.736178 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.736129    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:03.811433 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.811384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:03.865101 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.865071    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m" event={"ID":"298b5083-4edb-4714-96e4-a9d9a9a4a1e1","Type":"ContainerStarted","Data":"3e8f99e740f612dd1617487f3be23d75bc1234a0c95a7bc535e83a13e1fb0497"}
Nov 03 20:30:03.866542 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.866517    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68"}
Nov 03 20:30:03.867964 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.867925    2578 generic.go:356] "Generic (PLEG): container finished" podID="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" containerID="63bdaaef1c37182a92650028e2e86c1a245cb7502ef540180826edf132b82ee8" exitCode=0
Nov 03 20:30:03.868900 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.868098    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4ksx" event={"ID":"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735","Type":"ContainerDied","Data":"63bdaaef1c37182a92650028e2e86c1a245cb7502ef540180826edf132b82ee8"}
Nov 03 20:30:03.961761 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:03.961702    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056b04d8_72b3_4019_bddd_56a4469906de.slice/crio-139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe WatchSource:0}: Error finding container 139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe: Status 404 returned error can't find the container with id 139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe
Nov 03 20:30:03.963639 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.963605    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"]
Nov 03 20:30:03.985945 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:03.985888    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m" podStartSLOduration=2.060378141 podStartE2EDuration="3.985868486s" totalImagesPullingTime="1.925490345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.983147408 +0000 UTC m=+70.966368893" watchObservedRunningTime="2026-11-03 20:30:03.985868486 +0000 UTC m=+70.969089972"
Nov 03 20:30:04.821600 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:04.821568    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:04.834284 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:04.834256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:04.877229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:04.875744    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4ksx" event={"ID":"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735","Type":"ContainerStarted","Data":"5320e36d168768811d83d33860b7c7e29b39bba2e85c9c7ae5b9439dd5a23326"}
Nov 03 20:30:04.877229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:04.875806    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4ksx" event={"ID":"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735","Type":"ContainerStarted","Data":"25bf3c78d123b7a6efed4df4968178bb30486e60d4de5b67891f231d790eae3c"}
Nov 03 20:30:04.877229 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:04.877169    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe"}
Nov 03 20:30:05.080824 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.080711    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4ksx" podStartSLOduration=3.315849003 podStartE2EDuration="5.080684695s" totalImagesPullingTime="1.678118532s" totalInitContainerRuntime="86.71716ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:05.078962282 +0000 UTC m=+72.062183766" watchObservedRunningTime="2026-11-03 20:30:05.080684695 +0000 UTC m=+72.063906185"
Nov 03 20:30:05.092107 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.092041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:05.300671 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.299291    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"]
Nov 03 20:30:05.306791 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.306718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:05.321003 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.320905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.355804 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.355777    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"]
Nov 03 20:30:05.375949 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.375894    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w44xl\""
Nov 03 20:30:05.376066 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.375981    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 20:30:05.398610 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.398402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcba368e-649b-4dec-a6f3-edfb0f20ba38-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.398610 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.398514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj82j\" (UniqueName: \"kubernetes.io/projected/bcba368e-649b-4dec-a6f3-edfb0f20ba38-kube-api-access-zj82j\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.499574 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.499540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj82j\" (UniqueName: \"kubernetes.io/projected/bcba368e-649b-4dec-a6f3-edfb0f20ba38-kube-api-access-zj82j\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.499741 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.499582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcba368e-649b-4dec-a6f3-edfb0f20ba38-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.502734 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.502709    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcba368e-649b-4dec-a6f3-edfb0f20ba38-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.510035 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.510004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:05.562167 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.562137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj82j\" (UniqueName: \"kubernetes.io/projected/bcba368e-649b-4dec-a6f3-edfb0f20ba38-kube-api-access-zj82j\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.618564 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.618497    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-674d9b67d4-m4jmd"]
Nov 03 20:30:05.633920 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.633897    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.647241 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.647214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:05.677934 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.677892    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-674d9b67d4-m4jmd"]
Nov 03 20:30:05.688018 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.687993    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 20:30:05.688840 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.688789    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:30:05.701668 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-tls\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.701772 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-metrics-server-audit-profiles\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.701824 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-client-certs\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.701863 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0762ba86-bec7-4282-a042-ca63ffaeeea9-audit-log\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.701914 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtc8\" (UniqueName: \"kubernetes.io/projected/0762ba86-bec7-4282-a042-ca63ffaeeea9-kube-api-access-fmtc8\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.701967 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-client-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.702018 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.701991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.760399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.760234    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2str7q7cdaj8\""
Nov 03 20:30:05.760399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.760267    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jhfbw\""
Nov 03 20:30:05.760399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.760355    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 20:30:05.760399 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.760367    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 20:30:05.803169 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-tls\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.803282 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-metrics-server-audit-profiles\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.803282 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-client-certs\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.803282 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0762ba86-bec7-4282-a042-ca63ffaeeea9-audit-log\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.803418 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtc8\" (UniqueName: \"kubernetes.io/projected/0762ba86-bec7-4282-a042-ca63ffaeeea9-kube-api-access-fmtc8\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.803418 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-client-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.803418 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.803388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.804126 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.804100    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.804803 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.804773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0762ba86-bec7-4282-a042-ca63ffaeeea9-audit-log\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.805615 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.805593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-metrics-server-audit-profiles\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.810438 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.810405    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-client-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.812942 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.812917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-client-certs\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.813530 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.813507    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-tls\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.874223 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.874130    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtc8\" (UniqueName: \"kubernetes.io/projected/0762ba86-bec7-4282-a042-ca63ffaeeea9-kube-api-access-fmtc8\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:05.979845 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:05.979813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:06.063621 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.063590    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"]
Nov 03 20:30:06.142053 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:06.141980    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcba368e_649b_4dec_a6f3_edfb0f20ba38.slice/crio-fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea WatchSource:0}: Error finding container fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea: Status 404 returned error can't find the container with id fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea
Nov 03 20:30:06.393483 ip-10-0-135-141 kubenswrapper[2578]: W1103 20:30:06.393415    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0762ba86_bec7_4282_a042_ca63ffaeeea9.slice/crio-1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b WatchSource:0}: Error finding container 1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b: Status 404 returned error can't find the container with id 1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b
Nov 03 20:30:06.394003 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.393981    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-674d9b67d4-m4jmd"]
Nov 03 20:30:06.645320 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.645211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:06.847714 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.847680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:06.891916 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.891807    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nl7nq" event={"ID":"7574d083-15a3-4cf9-9547-93c53eb8222a","Type":"ContainerStarted","Data":"3610814eb5455b05b5e197f4f8cd9944cbfff949e4ca8be7fb7cd2012f998c86"}
Nov 03 20:30:06.893382 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.893325    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6" event={"ID":"bcba368e-649b-4dec-a6f3-edfb0f20ba38","Type":"ContainerStarted","Data":"fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea"}
Nov 03 20:30:06.895194 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.895166    2578 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed" exitCode=0
Nov 03 20:30:06.895319 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.895257    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed"}
Nov 03 20:30:06.896620 ip-10-0-135-141 kubenswrapper[2578]: I1103 20:30:06.896594    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd" event={"ID":"0762ba86-bec7-4282-a042-ca63ffaeeea9","Type":"ContainerStarted","Data":"1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b"}
Nov 03 20:30:06.947072 ip-10-0-135-141 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:30:06.962864 ip-10-0-135-141 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:30:06.963086 ip-10-0-135-141 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:30:06.963305 ip-10-0-135-141 systemd[1]: kubelet.service: Consumed 3.957s CPU time.
Nov 03 20:30:06.975130 ip-10-0-135-141 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:30:07.082710 ip-10-0-135-141 kubenswrapper[5318]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:30:07.082710 ip-10-0-135-141 kubenswrapper[5318]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:30:07.082710 ip-10-0-135-141 kubenswrapper[5318]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:30:07.083404 ip-10-0-135-141 kubenswrapper[5318]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:30:07.086139 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086055    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.086139 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086136    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.086139 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086140    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.086139 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086143    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.086139 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086146    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086149    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086152    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086155    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086158    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086160    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086163    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086166    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086168    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086171    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086174    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086177    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086180    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086191    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086194    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086197    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086209    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086213    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086216    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086219    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.086415 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086222    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086225    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086228    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086230    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086233    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086236    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086238    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086241    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086244    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086247    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086250    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086253    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086256    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086259    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086261    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086264    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086267    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086269    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086272    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086275    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.087087 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086277    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086280    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086283    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086286    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086288    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086291    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086294    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086297    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086299    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086302    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086305    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086307    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086310    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086312    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086316    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086321    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086324    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086328    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086331    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.087772 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086347    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086351    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086354    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086357    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086359    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086362    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086365    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086367    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086370    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086373    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086375    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086378    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086383    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086386    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086388    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086391    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086394    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086397    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086401    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.088369 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086405    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086409    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086414    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086418    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086423    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086428    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086431    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086434    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086437    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086439    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086442    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086446    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086450    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086454    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086457    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086460    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086462    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086465    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086468    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086471    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.089049 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086473    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086476    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086480    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.086483    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087299    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087307    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087311    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087314    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087317    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087320    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087323    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087326    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087329    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087331    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087350    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087353    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087356    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087359    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087362    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.089765 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087364    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087367    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087370    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087372    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087375    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087378    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087382    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087385    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087388    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087390    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087393    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087396    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087398    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087403    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087407    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087412    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087417    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087421    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087426    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087430    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.090452 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087434    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087439    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087444    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087449    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087453    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087458    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087462    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087466    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087470    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087474    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087478    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087482    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087485    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087488    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087491    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087493    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087496    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087499    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087502    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087505    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.091050 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087507    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087510    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087513    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087516    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087519    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087521    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087524    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087534    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087540    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087545    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087550    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087554    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087559    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087563    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087568    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087572    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087576    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087580    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087586    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.091865 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087591    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087595    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087599    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087606    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087612    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087616    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087618    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087621    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087623    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087626    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087629    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087632    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087634    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087637    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087640    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087642    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087645    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087648    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087650    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.092473 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087653    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087657    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087661    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087665    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087669    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087675    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087680    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087684    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087688    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087692    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087697    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087701    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.087705    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088287    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088294    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088298    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088301    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088303    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088306    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.093075 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088309    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088311    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088314    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088319    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088322    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088325    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088327    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088330    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088347    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088351    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088354    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088357    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088360    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088362    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088365    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088367    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088370    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088373    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088376    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.093714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088379    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088383    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088389    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088394    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088398    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088402    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088405    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088408    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088411    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088414    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088416    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088419    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088421    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088424    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088428    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088432    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088439    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088444    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088449    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.094371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088452    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088455    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088458    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088461    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088464    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088466    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088469    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088472    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088475    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088478    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088480    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088483    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088486    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088488    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088491    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088493    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088496    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088499    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088501    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088504    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.095069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088507    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088510    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088513    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088515    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088517    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088520    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088523    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088526    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088529    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088532    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088535    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088538    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088540    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088543    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088545    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088548    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088550    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088553    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088555    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.095786 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088558    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088562    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088566    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088570    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088576    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088580    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088583    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088585    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088588    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088591    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088593    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088596    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088599    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088601    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088604    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088608    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088611    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088613    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088616    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088618    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.096425 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088621    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088624    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.088629    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088755    5318 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088773    5318 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088782    5318 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088799    5318 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088808    5318 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088814    5318 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088820    5318 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088832    5318 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088835    5318 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088841    5318 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088844    5318 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088848    5318 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088851    5318 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088854    5318 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088857    5318 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088860    5318 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088863    5318 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088866    5318 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088871    5318 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088874    5318 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:30:07.097079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088878    5318 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088881    5318 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088884    5318 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088889    5318 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088892    5318 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088895    5318 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088899    5318 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088902    5318 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088905    5318 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088908    5318 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088911    5318 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088916    5318 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088921    5318 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088926    5318 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088931    5318 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088936    5318 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088940    5318 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088946    5318 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088952    5318 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088957    5318 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088962    5318 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088968    5318 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088973    5318 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088979    5318 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:30:07.097896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088983    5318 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088988    5318 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088993    5318 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.088997    5318 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089000    5318 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089003    5318 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089006    5318 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089009    5318 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089012    5318 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089015    5318 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089019    5318 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089022    5318 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089025    5318 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089028    5318 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089032    5318 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089131    5318 flags.go:64] FLAG: --help="false"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089135    5318 flags.go:64] FLAG: --hostname-override="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089140    5318 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089145    5318 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089153    5318 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089159    5318 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089163    5318 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089166    5318 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:30:07.098624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089169    5318 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089172    5318 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089175    5318 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089178    5318 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089181    5318 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089184    5318 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089189    5318 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089192    5318 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089195    5318 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089198    5318 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089201    5318 flags.go:64] FLAG: --lock-file=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089204    5318 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089207    5318 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089210    5318 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089216    5318 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089221    5318 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089226    5318 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089231    5318 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089234    5318 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089240    5318 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089245    5318 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089250    5318 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089258    5318 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089263    5318 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089271    5318 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:30:07.099263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089275    5318 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089278    5318 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089281    5318 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089285    5318 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089288    5318 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089291    5318 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089294    5318 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089301    5318 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089305    5318 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089308    5318 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089311    5318 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089314    5318 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089317    5318 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089321    5318 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089324    5318 flags.go:64] FLAG: --port="10250"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089327    5318 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089330    5318 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f6e8820218a2a23b"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089347    5318 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089352    5318 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089357    5318 flags.go:64] FLAG: --register-node="true"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089360    5318 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089364    5318 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089367    5318 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089369    5318 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089372    5318 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089376    5318 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:30:07.100056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089380    5318 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089383    5318 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089386    5318 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089389    5318 flags.go:64] FLAG: --runonce="false"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089394    5318 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089399    5318 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089413    5318 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089419    5318 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089424    5318 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089432    5318 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089438    5318 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089443    5318 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089448    5318 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089451    5318 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089454    5318 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089457    5318 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089460    5318 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089463    5318 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089466    5318 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089473    5318 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089476    5318 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089479    5318 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089485    5318 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089488    5318 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089491    5318 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:30:07.101032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089494    5318 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089498    5318 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089501    5318 flags.go:64] FLAG: --v="2"
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089506    5318 flags.go:64] FLAG: --version="false"
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089510    5318 flags.go:64] FLAG: --vmodule=""
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089514    5318 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.089518    5318 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089667    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089672    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089675    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089678    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089680    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089684    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089686    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089690    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089693    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089698    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089701    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089704    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089706    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089709    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.101800 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089713    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089717    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089721    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089726    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089729    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089736    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089740    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089744    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089748    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089752    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089756    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089759    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089762    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089764    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089767    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089770    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089773    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089775    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089778    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.102489 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089781    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089784    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089786    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089790    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089795    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089800    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089803    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089806    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089810    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089812    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089815    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089817    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089820    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089822    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089825    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089829    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089833    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089837    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089844    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.103269 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089848    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089852    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089856    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089860    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089864    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089868    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089872    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089875    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089877    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089880    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089882    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089885    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089888    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089891    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089893    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089896    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089898    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089901    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089903    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089906    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.104265 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089909    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089912    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089915    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089918    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089920    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089923    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089927    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089931    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089935    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089962    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089968    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089974    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089977    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089979    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089983    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089985    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089988    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089990    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089993    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089996    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.105112 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.089998    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090001    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090003    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090006    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090008    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090011    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090013    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090016    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090018    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090021    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090024    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090026    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090029    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090032    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.090076    5318 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090111    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090115    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.105936 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090118    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090122    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090125    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090128    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090130    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090133    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090137    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090140    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090142    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090145    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090147    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090150    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090153    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090156    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090158    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090161    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090163    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090166    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090169    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090172    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.106672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090175    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090180    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090184    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090188    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090192    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090196    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090207    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090212    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090216    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090220    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090225    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090228    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090231    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090234    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090238    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090241    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090244    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090246    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090251    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090254    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.107422 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090256    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090259    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090262    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090264    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090267    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090270    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090273    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090276    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090279    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090281    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090284    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090287    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090289    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090292    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090295    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090299    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090303    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090308    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090313    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.108013 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090317    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090320    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090323    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090326    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090349    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090352    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090355    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090360    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090363    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090365    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090368    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090372    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090375    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090377    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090380    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090384    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090388    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090392    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.108672 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090395    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090397    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090400    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090402    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090405    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090408    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090410    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090413    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090416    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090420    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090424    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090430    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090434    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090438    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090442    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090447    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090456    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090461    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090466    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090470    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.109248 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090475    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090480    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090484    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090488    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090493    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090498    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.090503    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.095644    5318 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.095666    5318 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.095690    5318 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.095697    5318 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095780    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095788    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095793    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095797    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095803    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095807    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095811    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095816    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.110076 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095821    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095825    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095830    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095834    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095838    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095843    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095847    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095851    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095856    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095860    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095865    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095869    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095873    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095878    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095882    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095886    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095890    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095895    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095899    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.110750 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095906    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095913    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095926    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095931    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095936    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095940    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095944    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095949    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095954    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095958    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095962    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095966    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095971    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095975    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095979    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095983    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095988    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095992    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.095996    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096000    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.111389 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096004    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096009    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096015    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096019    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096024    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096028    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096032    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096037    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096041    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096045    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096049    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096053    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096058    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096062    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096066    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096078    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096083    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096087    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096091    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.111986 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096095    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096099    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096106    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096112    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096117    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096122    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096127    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096131    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096135    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096140    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096144    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096148    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096153    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096157    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096162    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096166    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096170    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096174    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096178    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096183    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.112547 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096187    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096191    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096195    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096199    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096204    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096208    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096212    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096216    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096220    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096232    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096236    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096241    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096245    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096249    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096254    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096258    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096262    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096266    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096271    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.113159 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096275    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.096314    5318 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096600    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096611    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096617    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096622    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096627    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096633    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096638    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096643    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096648    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096652    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096657    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096661    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096665    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096669    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.113894 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096673    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096678    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096682    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096686    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096690    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096694    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096699    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096710    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096715    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096720    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096724    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096728    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096733    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096738    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096742    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096747    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096751    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096755    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096759    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.114595 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096763    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096767    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096771    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096776    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096780    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096784    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096788    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096792    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096796    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096800    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096804    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096808    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096812    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096816    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096821    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096825    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096829    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096833    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096837    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.115316 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096841    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096848    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096861    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096866    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096870    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096874    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096878    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096882    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096886    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096890    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096894    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096898    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096903    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096908    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096914    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096918    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096923    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096927    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096931    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096935    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.116007 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096939    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096943    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096949    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096953    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096964    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096969    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096973    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096977    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096981    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096985    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096989    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096993    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.096997    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097004    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097008    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097020    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097025    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097029    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097033    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.116646 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097037    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097041    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097046    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097050    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097054    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097059    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097063    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097068    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097072    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097076    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097080    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097084    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097089    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097093    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097099    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.097135    5318 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:30:07.117754 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097191    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097197    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097201    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097205    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097209    5318 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097214    5318 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097218    5318 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097222    5318 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097226    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097230    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097236    5318 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097240    5318 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097244    5318 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097256    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097261    5318 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097265    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097269    5318 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097274    5318 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097278    5318 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097282    5318 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:30:07.118362 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097286    5318 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097290    5318 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097295    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097299    5318 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097304    5318 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097308    5318 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097312    5318 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097317    5318 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097321    5318 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097325    5318 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097329    5318 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097347    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097352    5318 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097356    5318 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097361    5318 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097369    5318 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097374    5318 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097379    5318 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097383    5318 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097387    5318 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:30:07.119024 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097391    5318 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097395    5318 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097401    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097406    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097410    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097414    5318 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097425    5318 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097430    5318 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097435    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097440    5318 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097445    5318 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097449    5318 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097453    5318 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097457    5318 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097461    5318 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097465    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097469    5318 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097474    5318 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097478    5318 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:30:07.119667 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097482    5318 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097486    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097490    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097495    5318 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097502    5318 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097506    5318 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097510    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097514    5318 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097518    5318 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097522    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097526    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097531    5318 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097535    5318 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097539    5318 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097543    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097549    5318 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097554    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097561    5318 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097570    5318 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:30:07.120312 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097575    5318 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097586    5318 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097590    5318 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097595    5318 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097599    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097603    5318 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097607    5318 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097612    5318 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097617    5318 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097621    5318 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097625    5318 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097629    5318 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097634    5318 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097638    5318 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097642    5318 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097646    5318 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097650    5318 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097657    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097661    5318 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097665    5318 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:30:07.120945 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097670    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097674    5318 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097678    5318 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097682    5318 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097687    5318 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097691    5318 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097695    5318 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:07.097700    5318 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.097788    5318 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.099323    5318 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.099434    5318 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.099701    5318 server.go:1014] "Starting client certificate rotation"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.099814    5318 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.099888    5318 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:54 +0000 UTC" deadline="2028-07-23 15:27:03.909507492 +0000 UTC"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.099920    5318 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15066h56m56.809589229s"
Nov 03 20:30:07.121608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.100549    5318 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:30:07.122143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.102455    5318 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:30:07.122143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.107171    5318 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:30:07.122143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.109744    5318 log.go:25] "Validated CRI v1 image API"
Nov 03 20:30:07.122143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.110594    5318 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:30:07.122143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.112775    5318 fs.go:135] Filesystem UUIDs: map[70a88080-6513-41d6-849b-2122546f9e20:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a734207b-1850-4683-8a8a-f08da5fea8f9:/dev/nvme0n1p4]
Nov 03 20:30:07.122436 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.112795    5318 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0305c1b2-ad03-4e5b-847f-64b2828fedc2/volumes/kubernetes.io~projected/kube-api-access-c4r2z:{mountpoint:/var/lib/kubelet/pods/0305c1b2-ad03-4e5b-847f-64b2828fedc2/volumes/kubernetes.io~projected/kube-api-access-c4r2z major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0305c1b2-ad03-4e5b-847f-64b2828fedc2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0305c1b2-ad03-4e5b-847f-64b2828fedc2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~projected/kube-api-access-gh977:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~projected/kube-api-access-gh977 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~projected/kube-api-access-fmtc8:{mountpoint:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~projected/kube-api-access-fmtc8 major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90/volumes/kubernetes.io~projected/kube-api-access-6rgxb:{mountpoint:/var/lib/kubelet/pods/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90/volumes/kubernetes.io~projected/kube-api-access-6rgxb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12201904-e96b-4326-b87d-b5ac7539e37e/volumes/kubernetes.io~projected/kube-api-access-lxfnj:{mountpoint:/var/lib/kubelet/pods/12201904-e96b-4326-b87d-b5ac7539e37e/volumes/kubernetes.io~projected/kube-api-access-lxfnj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12201904-e96b-4326-b87d-b5ac7539e37e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/12201904-e96b-4326-b87d-b5ac7539e37e/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17928b13-2c16-48bd-9cad-e8920d246891/volumes/kubernetes.io~projected/kube-api-access-j2wql:{mountpoint:/var/lib/kubelet/pods/17928b13-2c16-48bd-9cad-e8920d246891/volumes/kubernetes.io~projected/kube-api-access-j2wql major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19efb517-1936-48f3-ab42-bd7ee7a87239/volumes/kubernetes.io~projected/kube-api-access-tpbtk:{mountpoint:/var/lib/kubelet/pods/19efb517-1936-48f3-ab42-bd7ee7a87239/volumes/kubernetes.io~projected/kube-api-access-tpbtk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~projected/kube-api-access-48rd6:{mountpoint:/var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~projected/kube-api-access-48rd6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~projected/kube-api-access-rrv6d:{mountpoint:/var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~projected/kube-api-access-rrv6d major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c38cada-a1d9-40a5-83c2-ac7157fb9975/volumes/kubernetes.io~projected/kube-api-access-jqgf4:{mountpoint:/var/lib/kubelet/pods/2c38cada-a1d9-40a5-83c2-ac7157fb9975/volumes/kubernetes.io~projected/kube-api-access-jqgf4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~projected/kube-api-access-b95rw:{mountpoint:/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~projected/kube-api-access-b95rw major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~projected/kube-api-access-hchxh:{mountpoint:/var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~projected/kube-api-access-hchxh major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~projected/kube-api-access-6bdbx:{mountpoint:/var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~projected/kube-api-access-6bdbx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48361014-a0f8-4050-8fdf-bf7d7b70ddcb/volumes/kubernetes.io~projected/kube-api-access-mcmww:{mountpoint:/var/lib/kubelet/pods/48361014-a0f8-4050-8fdf-bf7d7b70ddcb/volumes/kubernetes.io~projected/kube-api-access-mcmww major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48361014-a0f8-4050-8fdf-bf7d7b70ddcb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/48361014-a0f8-4050-8fdf-bf7d7b70ddcb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bce3daf-4de1-4fea-89e8-cd891e432355/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4bce3daf-4de1-4fea-89e8-cd891e432355/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee/volumes/kubernetes.io~projected/kube-api-access-s8cph:{mountpoint:/var/lib/kubelet/pods/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee/volumes/kubernetes.io~projected/kube-api-access-s8cph major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7574d083-15a3-4cf9-9547-93c53eb8222a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7574d083-15a3-4cf9-9547-93c53eb8222a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volumes/kubernetes.io~projected/kube-api-access-qzmkw:{mountpoint:/var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volumes/kubernetes.io~projected/kube-api-access-qzmkw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9285d5c2-ae97-46ad-974a-2e2cbce9b36e/volumes/kubernetes.io~projected/kube-api-access-jxlbz:{mountpoint:/var/lib/kubelet/pods/9285d5c2-ae97-46ad-974a-2e2cbce9b36e/volumes/kubernetes.io~projected/kube-api-access-jxlbz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9285d5c2-ae97-46ad-974a-2e2cbce9b36e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9285d5c2-ae97-46ad-974a-2e2cbce9b36e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~empty-dir/config-out major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~projected/kube-api-access-g4lqq:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~projected/kube-api-access-g4lqq major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~projected/tls-assets major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/config-volume major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/web-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a641934f-d459-46af-be40-6aee398f3b82/volumes/kubernetes.io~projected/kube-api-access-vjr6x:{mountpoint:/var/lib/kubelet/pods/a641934f-d459-46af-be40-6aee398f3b82/volumes/kubernetes.io~projected/kube-api-access-vjr6x major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcba368e-649b-4dec-a6f3-edfb0f20ba38/volumes/kubernetes.io~projected/kube-api-access-zj82j:{mountpoint:/var/lib/kubelet/pods/bcba368e-649b-4dec-a6f3-edfb0f20ba38/volumes/kubernetes.io~projected/kube-api-access-zj82j major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcba368e-649b-4dec-a6f3-edfb0f20ba38/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bcba368e-649b-4dec-a6f3-edfb0f20ba38/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7871e65-e639-4c12-ad40-4a2dd1b90687/volumes/kubernetes.io~projected/kube-api-access-82x82:{mountpoint:/var/lib/kubelet/pods/c7871e65-e639-4c12-ad40-4a2dd1b90687/volumes/kubernetes.io~projected/kube-api-access-82x82 major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c61afc3666c9e050fa8399d01a0e7775f3ebd97cdcdd8262500cb88c651b236d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/07773d741bbc5907ebca9147b2fff0047032aedec844398ec1188466ac325f02/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3c22c542c8923a67c0fa8a3ed0bb4d51866fffe2d109833d4314a94878fa947c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/3bd40eef17155a18f3d5e6df10bf1831f9e0f00a6d306b4b74a56853ef13db25/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c6449e8aeb1003bbb53798040561fc3587c2c68f0d53eacda30a84ad38053d7b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/9091b788ea94b54eab1ae09dcb7beb82f22209317018bd5922e8bde97bc0ca6e/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/93c319d3116aa034943767864462c1c217d1567dbebb8a8ea3356c8c6572882a/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/45acc381c5c2723670360eb22736a36475fa5d53f798312c4f0c8dd250924533/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/36e109a7ebcf03829dce6edcc2087434412c6fa505ac56e2502b94bfadc1a48c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ff547070362526c19b70c8672080dae4dad85577755544f29d2588923270fe58/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c3d731514a84e83eaef5a97862f94618f40943211b47db55cffd3dd95ad1da0d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2e23d2231ff1a94617ae9aeebb2c510260c18a30bf753f37c754dc25ac530a01/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0a08c4d310445343790ef1d556da305fcc63cc7045236fe0a17a2a15dd5ffde2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/101ef761dc571fa85c5f82d12b350c7b5fba2165d9430d5f0ec7051a38adf698/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/dcb5b6f4e76cd4603bdd5c24e3641e6ff3ad134b225fbb0b1c2044ccfa4a0f16/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/fae3be1acae5c7c12b8ef9722117a4615746575689dc4183358509cf350a40d2/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/8d45e0d17734700d0344e72f058acb2787fa2de761e678235ef55347abe7cd09/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/f9e7b73afd0b598fd1e2a9e40fc86fd93327d505855a51d3d131f6b1f6cbaa5a/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/75aa7d77ff4fe0312dfdd2b1cf1894261417e908b414712f9b94fea07b5181f5/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/9bf351e229420cd7b5cfd8adcb5cad12d4178ed6b45baadc9713c0a7fe06a135/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/224a7f796bf9dad79e5cacfca505607fb793819114b0cd3c75df2cd1e51aee1d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/03316818cd850f92c8ba84827491ffdd78fce72cd50d5edcb1c6829d13016a04/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2b313eaa946a5db438823b0f6e4144903273254ff6579313b931a624d9c34867/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/32d440b7241855755326a32729d4420e4e02a694a8a09344b19ad8fdaf42da42/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/116472c0e047ea646b7e51c6e11dd3949108bfd4c4a91223f28bd7c9c874dc67/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/3ccdfd3608e1008ab133c585334a283a85c226e94b35776a17b0bb3037f5ebb5/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/c2049df4ad2a0d5ccbe9b00d1d7c1377f28b7b6f2e58332e8c98311fc0163d01/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/4ee11f6298b1690f4a58ac91cb6c7f4ce91dd7483f8d41b30e39f3e95bcb3169/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/36bd3747bd5fdb720b5664ef0028c780125f20292b49664c978f6a842e062d0a/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b801a4bc46b4081818b3bbe20f1286b0191fdc59713bdc0570075b1977d71c9e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/3ff4790a7780c505455635e62d66c1a72484f27a7e3da4fb30ea2c2a2e31b9ff/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/d03f20e159bc884fa12f6989c648f8d224282bc4d5a47ad045644fbdc55030c2/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/9221d62cbfdf05cf8793d24c64a1e9ab8db5b20fb1465835ebcd1a5acd0bb183/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/37cf3f76e9100d3eb68157f56c66fb96470a8854711dff58ddc64532073e06ea/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/7f8d17fb5702fea53a5c785b025a81df97a2e91a1f1c125ea6e0cd9e413c46e4/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/3675697085d2e1ab14d3463ae97b3d2c13e3a69ad0ac2fc0a58f89c9ed5b62e5/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/194335d5801fb191a3680f4c4e1f594cf51eb0583898bef710071e50c5c4c274/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4b9555d83ae327f04e576f0fb29736c8f00430bd7aea0c0fb49a1a6c29c11133/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/0daafb65f9e92a3b83b7f09c7acd343bc79e2fa9c46a5016294083df3ef0bbe7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/10af14977dcd2d1579fcb67c3930f95fbb174fc447d8b78f2b7739de734b791f/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/3338e06550fedde59559c5f07c09dee96b01c05bcddb4e961c6dbde97ccb5f0d/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/82d2b014f8a71438063a02e11cb9e80d1f8f379bad71a1543daaec41af00c6d9/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/bd5d3618f2f9af5b021c68b4cdb40fe9f9bf14c2d68dffdba611cb56c4fe830e/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/d35f2cce557ac7669f56a8b67b8bf0e17b7a66944edd449a228eea17856a3194/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/0f2b88d2e530ecb67b9d1efb691c7e575543b496d5bdf8959a9f9b3cb4524e64/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/19c37b34eb56caa74ca4d506985a51acf7d0773091e1d62d00f247bc626ec1fd/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/938e5374b6f1b33ce3b897cdd645ccb2b438a093497d2847389d893912efe1f2/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/fbc4db4fb9bdee3a4214abfaa8992f3ec2f1b2943f52dd7a0b90f85a30c1ba43/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/3516059e39fbac7d0784f4057ace07caa84c1f60a7b8e08d01442c01ef0422fd/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/7edbab1d57bf637abdcaf2644c68ee58b1038a67a56b468d1b990f88b36b417d/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/64b264dd19ca045a93102c48b7af2a7c0f5f4b3397c89491e3cb01340b575cbf/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/bd6c5eb9a36f3445db658064ea77c7860c2d9eec9700776bea8f449956a6f3d4/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e52a561235a2f9fa82c2f4caaae5dcd506fc2a80271776254039b72d5ee9408f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b4949b91616625732fb77683f0ebb81f4e11842219bc724c99b6897c4ee96e95/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9892390c73593aa8f6df6c40088a66e73ac8abf17e694a0342442f5c7d2fefe8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/51c85206cb61f9d1f070ee4d619989ce7c0e1c2458514aafbb687695ad0a5393/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c0121a194a61966ac0296d90cf373c0215798340a32c058b5426dd577ffaf0c5/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a53e6bcce22c49598f28e29131826da207f7a9339695db4b3252609a54dd3e07/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/02c018cae7234d6d3966f9c046c6b3eb57a679ddde65429df1c1cadf32c9279c/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b8f6882209207944faa924ff23e2bde688d9113a9f91aa12012cb6b165531fe9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/885cafa76c040385eacfaf32ebbf90670f15cb03e9cd90ae0d7a6403225dd036/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3a2560ba28043237214406764c7f672961412d893194b4bdb3bb4107dc7c8dcc/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 20:30:07.122886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.122415    5318 manager.go:217] Machine: {Timestamp:2026-11-03 20:30:07.121613935 +0000 UTC m=+0.101170766 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100720 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec207adaec32ed2e14dff9b23eff26b7 SystemUUID:ec207ada-ec32-ed2e-14df-f9b23eff26b7 BootID:f4d7bd73-fca4-48d2-8ef9-04dea88c8ee6 Filesystems:[{Device:/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c38cada-a1d9-40a5-83c2-ac7157fb9975/volumes/kubernetes.io~projected/kube-api-access-jqgf4 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:351 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c7871e65-e639-4c12-ad40-4a2dd1b90687/volumes/kubernetes.io~projected/kube-api-access-82x82 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8d72de4414e0eea1b02358adf6d39d1617a39db29ae03fb683b373f2fa69e1c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12201904-e96b-4326-b87d-b5ac7539e37e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a1c534a277b29d4f362bf190adf86a425c3b0ead3bcefd548fb4a980e2ec0fb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bcba368e-649b-4dec-a6f3-edfb0f20ba38/volumes/kubernetes.io~projected/kube-api-access-zj82j DeviceMajor:0 DeviceMinor:373 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volumes/kubernetes.io~projected/kube-api-access-qzmkw DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~projected/kube-api-access-6bdbx DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~projected/kube-api-access-rrv6d DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bcba368e-649b-4dec-a6f3-edfb0f20ba38/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7af0fdc0a126a20409f9d3b8fc5d7bcc6912197597c7370782283dfc1d505bd/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9971f3b03b586482349269cf456083562dede65dde1fbff13aa2ba9b9ac33c8b/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19efb517-1936-48f3-ab42-bd7ee7a87239/volumes/kubernetes.io~projected/kube-api-access-tpbtk DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43dca4b39b9d7074d128a1a7a849569cea894d0c0baca2fe8d2e43ec34e7ffb0/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12201904-e96b-4326-b87d-b5ac7539e37e/volumes/kubernetes.io~projected/kube-api-access-lxfnj DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9285d5c2-ae97-46ad-974a-2e2cbce9b36e/volumes/kubernetes.io~projected/kube-api-access-jxlbz DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59aa87ff9d9ca9cf38df6c8ae3a658a4c69979252958160b1888ca51beeb2b59/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2acb87b67054fb25d4af1888d05b6fe2f601c04144466804655e04b950db7915/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90/volumes/kubernetes.io~projected/kube-api-access-6rgxb DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/volumes/kubernetes.io~projected/kube-api-access-hchxh DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cdac693-b083-4a91-bd5c-ea56d23bd771/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0305c1b2-ad03-4e5b-847f-64b2828fedc2/volumes/kubernetes.io~projected/kube-api-access-c4r2z DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0762ba86-bec7-4282-a042-ca63ffaeeea9/volumes/kubernetes.io~projected/kube-api-access-fmtc8 DeviceMajor:0 DeviceMinor:379 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb7d1055a3946616037f166d5a07a43ae3d2647d2bea449604726fcd44667b14/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0305c1b2-ad03-4e5b-847f-64b2828fedc2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~projected/kube-api-access-gh977 DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad43bc7a0421ca21cefd820615bf0873ca0b56f39791a9cdd728105aa04189f1/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29251496f3e4c4778712e6998f73943d21fd6980029e1872e6b8f5c92329bf2b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes/kubernetes.io~projected/kube-api-access-b95rw DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/298b5083-4edb-4714-96e4-a9d9a9a4a1e1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~projected/kube-api-access-g4lqq DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42f1ee76-96ba-4983-b33a-b7321b678045/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ffbb54dde4577402f4f6096b3934ca18a89298325fbe304d3726f4c0b3a8c7e/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fecf9f2c1e029f469003f20c718b843f9e171d54a40eb77979dae1cc8f5379ea/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c835eb6cc594a1317aba0ba3dffcac9063e188e9bf8381f0707b46a34b159c5/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d446b2cdb75ad9ac16fdd102120919c64affb4d882fa4a55f9ba8c9317956532/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee/volumes/kubernetes.io~projected/kube-api-access-s8cph DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48361014-a0f8-4050-8fdf-bf7d7b70ddcb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bce3daf-4de1-4fea-89e8-cd891e432355/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7574d083-15a3-4cf9-9547-93c53eb8222a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/139b36a93c001491ba64b5f8a373e31b51b8e2b5febf95fabcdd64de4aa341fe/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e460c95540ca9ea7df172388528464f77d2f565b5100ae4e11d7d3da530484fb/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/157d8f9eed6a7b337fbda99fb1d7a9ce27bebade3e72ca35dd56d840a48e3a03/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e377adddfb3bcb50a8096c5916ffb024f1b60ffe1a419d8c1b58831804e6f7b/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15d1b0cf1e3405ef2d725deeabe46d827bf61d9b06403cd3f46baeb295a3deda/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/volumes/kubernetes.io~projected/kube-api-access-48rd6 DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48361014-a0f8-4050-8fdf-bf7d7b70ddcb/volumes/kubernetes.io~projected/kube-api-access-mcmww DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b59460258e7b4bfed3cf8252ca38f9c6a677e730a05ea437dbbd2fd222d6d86/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17928b13-2c16-48bd-9cad-e8920d246891/volumes/kubernetes.io~projected/kube-api-access-j2wql DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9285d5c2-ae97-46ad-974a-2e2cbce9b36e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/056b04d8-72b3-4019-bddd-56a4469906de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a641934f-d459-46af-be40-6aee398f3b82/volumes/kubernetes.io~projected/kube-api-access-vjr6x DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:139b36a93c00149 MacAddress:52:60:d9:30:41:3f Speed:10000 Mtu:8901} {Name:157d8f9eed6a7b3 MacAddress:d6:b9:3f:82:bc:1e Speed:10000 Mtu:8901} {Name:15d1b0cf1e3405e MacAddress:d6:73:e1:7f:b6:2b Speed:10000 Mtu:8901} {Name:1e377adddfb3bcb MacAddress:9a:e7:6f:cf:b3:aa Speed:10000 Mtu:8901} {Name:2acb87b67054fb2 MacAddress:02:79:f3:84:cf:92 Speed:10000 Mtu:8901} {Name:3b59460258e7b4b MacAddress:8e:fd:01:5e:21:16 Speed:10000 Mtu:8901} {Name:3ffbb54dde45774 MacAddress:6e:9f:10:ed:a1:de Speed:10000 Mtu:8901} {Name:7c835eb6cc594a1 MacAddress:82:b5:bd:05:e9:2c Speed:10000 Mtu:8901} {Name:9eaa0d8f93e0ce6 MacAddress:7a:ac:db:1c:79:d4 Speed:10000 Mtu:8901} {Name:b534ead6333a7ac MacAddress:f6:1c:15:f0:75:92 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f5:46:59:1d Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:70:d0:99:0a:52 Speed:0 Mtu:8901} {Name:e460c95540ca9ea MacAddress:22:f9:05:21:a6:e6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f5:46:59:1d Speed:0 Mtu:9001} {Name:f7af0fdc0a126a2 MacAddress:56:25:96:d6:e4:4d Speed:10000 Mtu:8901} {Name:fecf9f2c1e029f4 MacAddress:4a:d3:4c:a3:c3:8e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2a:03:2f:66:38:f5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:72:26:f4:d4:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:30:07.123115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.122891    5318 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:30:07.123115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.122939    5318 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:30:07.123289 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123266    5318 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:30:07.123476 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123290    5318 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-141.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:30:07.123538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123480    5318 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:30:07.123538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123491    5318 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:30:07.123538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123506    5318 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:30:07.123538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123530    5318 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:30:07.123679 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123612    5318 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:30:07.123812 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123801    5318 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:30:07.123940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123930    5318 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:30:07.123975 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123967    5318 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:30:07.124019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123975    5318 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:30:07.124019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.123987    5318 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:30:07.124019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.124001    5318 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:30:07.124019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.124014    5318 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:30:07.125182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.125163    5318 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:30:07.125182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.125181    5318 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:30:07.126517 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.126491    5318 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:30:07.211441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.211407    5318 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:30:07.211441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.211430    5318 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:30:07.211691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.211579    5318 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:30:07.211857 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.211841    5318 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:30:07.211940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.211883    5318 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:30:07.212025 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212014    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:30:07.212087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212034    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:30:07.212087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212045    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:30:07.212087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212054    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:30:07.212087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212063    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:30:07.212087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212073    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:30:07.212087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212082    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212092    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212103    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212112    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212126    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212139    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212173    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:30:07.212431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.212182    5318 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:30:07.215962 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.215944    5318 server.go:1263] "Started kubelet"
Nov 03 20:30:07.216447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.216016    5318 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:30:07.216447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.216057    5318 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:30:07.216447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.216139    5318 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:30:07.216648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.216471    5318 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:30:07.216591 ip-10-0-135-141 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:30:07.217904 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.217466    5318 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.219572    5318 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.219587    5318 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.219640    5318 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.219678    5318 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:56 +0000 UTC" deadline="2028-06-25 06:43:01.165241635 +0000 UTC"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.219744    5318 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14386h12m53.945500709s"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.219789    5318 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:30:07.219929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.219898    5318 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:30:07.220201 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.220013    5318 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:30:07.220924 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.220902    5318 factory.go:55] Registering systemd factory
Nov 03 20:30:07.220924 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.220926    5318 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:30:07.221422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.221404    5318 factory.go:153] Registering CRI-O factory
Nov 03 20:30:07.221516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.221429    5318 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:30:07.221516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.221488    5318 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:30:07.221516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.221507    5318 factory.go:103] Registering Raw factory
Nov 03 20:30:07.221678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.221525    5318 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.221960    5318 manager.go:319] Starting recovery of all containers
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.223867    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a641934f-d459-46af-be40-6aee398f3b82" volumeName="kubernetes.io/configmap/a641934f-d459-46af-be40-6aee398f3b82-serviceca" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.223925    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0305c1b2-ad03-4e5b-847f-64b2828fedc2" volumeName="kubernetes.io/secret/0305c1b2-ad03-4e5b-847f-64b2828fedc2-metrics-tls" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.223984    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.223999    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17928b13-2c16-48bd-9cad-e8920d246891" volumeName="kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224020    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298b5083-4edb-4714-96e4-a9d9a9a4a1e1" volumeName="kubernetes.io/projected/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-kube-api-access-rrv6d" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224052    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298b5083-4edb-4714-96e4-a9d9a9a4a1e1" volumeName="kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224086    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298b5083-4edb-4714-96e4-a9d9a9a4a1e1" volumeName="kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224111    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdac693-b083-4a91-bd5c-ea56d23bd771" volumeName="kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224127    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224141    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224152    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224186    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224217    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-client-ca-bundle" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224240    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19efb517-1936-48f3-ab42-bd7ee7a87239" volumeName="kubernetes.io/projected/19efb517-1936-48f3-ab42-bd7ee7a87239-kube-api-access-tpbtk" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224256    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" volumeName="kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224266    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48361014-a0f8-4050-8fdf-bf7d7b70ddcb" volumeName="kubernetes.io/configmap/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224287    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bce3daf-4de1-4fea-89e8-cd891e432355" volumeName="kubernetes.io/secret/4bce3daf-4de1-4fea-89e8-cd891e432355-agent-certs" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224306    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/configmap/056b04d8-72b3-4019-bddd-56a4469906de-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:30:07.224361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224331    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224380    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/projected/0762ba86-bec7-4282-a042-ca63ffaeeea9-kube-api-access-fmtc8" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224425    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48361014-a0f8-4050-8fdf-bf7d7b70ddcb" volumeName="kubernetes.io/empty-dir/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-data-volume" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224437    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224458    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224473    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224521    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7871e65-e639-4c12-ad40-4a2dd1b90687" volumeName="kubernetes.io/projected/c7871e65-e639-4c12-ad40-4a2dd1b90687-kube-api-access-82x82" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224892    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.224965    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17928b13-2c16-48bd-9cad-e8920d246891" volumeName="kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225003    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298b5083-4edb-4714-96e4-a9d9a9a4a1e1" volumeName="kubernetes.io/configmap/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225030    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f1ee76-96ba-4983-b33a-b7321b678045" volumeName="kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-tmp" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225054    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/empty-dir/0762ba86-bec7-4282-a042-ca63ffaeeea9-audit-log" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225068    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" volumeName="kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225089    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f1ee76-96ba-4983-b33a-b7321b678045" volumeName="kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-etc-tuned" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225109    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f1e1e3-49b2-4665-b2a7-5e0e807d0cee" volumeName="kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225123    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdac693-b083-4a91-bd5c-ea56d23bd771" volumeName="kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-env-overrides" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225142    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12201904-e96b-4326-b87d-b5ac7539e37e" volumeName="kubernetes.io/projected/12201904-e96b-4326-b87d-b5ac7539e37e-kube-api-access-lxfnj" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225159    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42f1ee76-96ba-4983-b33a-b7321b678045" volumeName="kubernetes.io/projected/42f1ee76-96ba-4983-b33a-b7321b678045-kube-api-access-6bdbx" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225176    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48361014-a0f8-4050-8fdf-bf7d7b70ddcb" volumeName="kubernetes.io/secret/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 20:30:07.225187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225191    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9285d5c2-ae97-46ad-974a-2e2cbce9b36e" volumeName="kubernetes.io/projected/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-kube-api-access-jxlbz" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225205    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225232    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcba368e-649b-4dec-a6f3-edfb0f20ba38" volumeName="kubernetes.io/projected/bcba368e-649b-4dec-a6f3-edfb0f20ba38-kube-api-access-zj82j" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225251    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-grpc-tls" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225273    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cae6ab2-e17e-4b19-8e99-f36f7072a3b7" volumeName="kubernetes.io/projected/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-kube-api-access-hchxh" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225289    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7574d083-15a3-4cf9-9547-93c53eb8222a" volumeName="kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225302    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225321    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225363    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225386    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cae6ab2-e17e-4b19-8e99-f36f7072a3b7" volumeName="kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225404    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0305c1b2-ad03-4e5b-847f-64b2828fedc2" volumeName="kubernetes.io/projected/0305c1b2-ad03-4e5b-847f-64b2828fedc2-kube-api-access-c4r2z" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225419    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0be5ee45-4c57-4f1f-a3a1-075c2dca3c90" volumeName="kubernetes.io/empty-dir/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-tmp-dir" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225438    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17928b13-2c16-48bd-9cad-e8920d246891" volumeName="kubernetes.io/projected/17928b13-2c16-48bd-9cad-e8920d246891-kube-api-access-j2wql" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225457    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c38cada-a1d9-40a5-83c2-ac7157fb9975" volumeName="kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225559    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c38cada-a1d9-40a5-83c2-ac7157fb9975" volumeName="kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225573    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225585    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225603    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225616    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0305c1b2-ad03-4e5b-847f-64b2828fedc2" volumeName="kubernetes.io/configmap/0305c1b2-ad03-4e5b-847f-64b2828fedc2-config-volume" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225703    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225721    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225736    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdac693-b083-4a91-bd5c-ea56d23bd771" volumeName="kubernetes.io/projected/8cdac693-b083-4a91-bd5c-ea56d23bd771-kube-api-access-qzmkw" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225753    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdac693-b083-4a91-bd5c-ea56d23bd771" volumeName="kubernetes.io/secret/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225767    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225783    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225823    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a641934f-d459-46af-be40-6aee398f3b82" volumeName="kubernetes.io/projected/a641934f-d459-46af-be40-6aee398f3b82-kube-api-access-vjr6x" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225835    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0305c1b2-ad03-4e5b-847f-64b2828fedc2" volumeName="kubernetes.io/empty-dir/0305c1b2-ad03-4e5b-847f-64b2828fedc2-tmp-dir" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225902    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" volumeName="kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.225941    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226021    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cae6ab2-e17e-4b19-8e99-f36f7072a3b7" volumeName="kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226042    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48361014-a0f8-4050-8fdf-bf7d7b70ddcb" volumeName="kubernetes.io/projected/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-api-access-mcmww" seLinuxMountContext=""
Nov 03 20:30:07.226105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226118    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7871e65-e639-4c12-ad40-4a2dd1b90687" volumeName="kubernetes.io/configmap/c7871e65-e639-4c12-ad40-4a2dd1b90687-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226142    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12201904-e96b-4326-b87d-b5ac7539e37e" volumeName="kubernetes.io/secret/12201904-e96b-4326-b87d-b5ac7539e37e-cert" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226158    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" volumeName="kubernetes.io/empty-dir/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-textfile" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226180    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" volumeName="kubernetes.io/projected/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-kube-api-access-48rd6" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226215    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226231    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c38cada-a1d9-40a5-83c2-ac7157fb9975" volumeName="kubernetes.io/projected/2c38cada-a1d9-40a5-83c2-ac7157fb9975-kube-api-access-jqgf4" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226286    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226299    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226318    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226331    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcba368e-649b-4dec-a6f3-edfb0f20ba38" volumeName="kubernetes.io/secret/bcba368e-649b-4dec-a6f3-edfb0f20ba38-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226407    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0be5ee45-4c57-4f1f-a3a1-075c2dca3c90" volumeName="kubernetes.io/projected/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-kube-api-access-6rgxb" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226426    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336018cd-e7cc-4f23-809a-8b67aebd020d" volumeName="kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226480    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bce3daf-4de1-4fea-89e8-cd891e432355" volumeName="kubernetes.io/configmap/4bce3daf-4de1-4fea-89e8-cd891e432355-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226524    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdac693-b083-4a91-bd5c-ea56d23bd771" volumeName="kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-config" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226551    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/projected/056b04d8-72b3-4019-bddd-56a4469906de-kube-api-access-gh977" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226565    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0762ba86-bec7-4282-a042-ca63ffaeeea9" volumeName="kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226606    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b04d8-72b3-4019-bddd-56a4469906de" volumeName="kubernetes.io/empty-dir/056b04d8-72b3-4019-bddd-56a4469906de-tmp" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226619    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17928b13-2c16-48bd-9cad-e8920d246891" volumeName="kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226657    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" volumeName="kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-tls" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226671    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cae6ab2-e17e-4b19-8e99-f36f7072a3b7" volumeName="kubernetes.io/configmap/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226684    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9285d5c2-ae97-46ad-974a-2e2cbce9b36e" volumeName="kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226703    5318 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" volumeName="kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out" seLinuxMountContext=""
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226738    5318 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:30:07.227616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.226746    5318 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:30:07.246142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.246103    5318 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:30:07.247445 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.247424    5318 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:30:07.247524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.247450    5318 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:30:07.247524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.247504    5318 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:30:07.247634 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.247568    5318 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:30:07.271957 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.271923    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:30:07.272346 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.272303    5318 generic.go:356] "Generic (PLEG): container finished" podID="8cdac693-b083-4a91-bd5c-ea56d23bd771" containerID="738ac27585078545917c948af9caba3367bf6704c07cf73ca05ef14033d2d355" exitCode=1
Nov 03 20:30:07.276652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.276625    5318 generic.go:356] "Generic (PLEG): container finished" podID="ba15cf602964f30d9df06d36baaae3a5" containerID="796e1aebe4685889ab7430fa65c3a69062e94c71ee6502294df09a8eb67a4705" exitCode=0
Nov 03 20:30:07.280835 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.280808    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed" exitCode=0
Nov 03 20:30:07.283680 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.283655    5318 generic.go:356] "Generic (PLEG): container finished" podID="1af07fe5-98e6-4f65-a70f-1b5cfe9c1735" containerID="63bdaaef1c37182a92650028e2e86c1a245cb7502ef540180826edf132b82ee8" exitCode=0
Nov 03 20:30:07.287889 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.287871    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:30:07.288737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.288715    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:30:07.294506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.294484    5318 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="acc8cc27df731dfa6a70c9794d196df165256ee20df553e158fbd0814f71cc2e" exitCode=0
Nov 03 20:30:07.294506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.294502    5318 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="f8f44f1e7e523de89a1644ee5bb04c60d41eb2ba6bd3c03038831878f845c0c2" exitCode=0
Nov 03 20:30:07.294506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.294509    5318 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="563fc0290f216d3fcb93ea337fd75bf7737e39ab34d3b00952cb96a4b7591cb1" exitCode=0
Nov 03 20:30:07.294708 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.294517    5318 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="4e4149f799c81b7d6743b3c63387f2dd14ee28b2ea4af729ddc39367ff6fa033" exitCode=0
Nov 03 20:30:07.294708 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.294524    5318 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="a5a13023c3331416ea76819bc0d3970d40fe4fe35c992885cd75639c8ddf6d0f" exitCode=0
Nov 03 20:30:07.294708 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.294530    5318 generic.go:356] "Generic (PLEG): container finished" podID="17928b13-2c16-48bd-9cad-e8920d246891" containerID="2b6a1623b7102203df31717183b412182c565e5a08457494747e5d9c30f47688" exitCode=0
Nov 03 20:30:07.319873 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.319850    5318 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:30:07.321397 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.321383    5318 manager.go:324] Recovery completed
Nov 03 20:30:07.322415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.322397    5318 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:30:07.323306 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.323288    5318 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:30:07.340091 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.340074    5318 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:30:07.342772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.342754    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:30:07.342833 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.342787    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:30:07.342833 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.342800    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:30:07.345178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345158    5318 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:30:07.345178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345174    5318 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:30:07.345313 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345174    5318 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:30:07.345313 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345192    5318 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:30:07.345406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345331    5318 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:30:07.345406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345362    5318 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:30:07.345406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345383    5318 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:30:07.345406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345388    5318 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:30:07.345406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345394    5318 policy_none.go:50] "Start"
Nov 03 20:30:07.345406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345403    5318 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:30:07.345569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345413    5318 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:30:07.345569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345543    5318 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:30:07.345569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345550    5318 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:30:07.345569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.345555    5318 policy_none.go:44] "Start"
Nov 03 20:30:07.346158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.346143    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:30:07.346214 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.346176    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:30:07.346214 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.346192    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:30:07.347758 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.347735    5318 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 20:30:07.350070 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.350046    5318 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:30:07.352922 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.352910    5318 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:30:07.353522 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.352932    5318 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:30:07.353563 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.353525    5318 server.go:87] "Starting device plugin registration server"
Nov 03 20:30:07.353769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.353758    5318 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:30:07.353806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.353770    5318 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:30:07.353879 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.353852    5318 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:30:07.354012 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.353935    5318 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:30:07.354099 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.354037    5318 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:30:07.354099 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.354045    5318 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:30:07.354193 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.354093    5318 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:30:07.354024849Z","UUID":"3d64290a-da5c-4652-9d9f-41b567534582","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:30:07.355533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.355474    5318 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:30:07.355533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.355499    5318 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:30:07.355796 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.355756    5318 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:30:07.355796 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.355791    5318 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-141.us-west-2.compute.internal\" not found"
Nov 03 20:30:07.371250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.371234    5318 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:30:07.371250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.371253    5318 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:30:07.415290 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.415275    5318 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:30:07.454616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.454598    5318 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:30:07.455476 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.455454    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:30:07.455611 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.455479    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:30:07.455611 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.455490    5318 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-141.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:30:07.455611 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.455543    5318 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.548097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.548029    5318 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"]
Nov 03 20:30:07.549800 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.549748    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ebc5d4efd8a63d505e3fa8dc27ffb1de","Type":"ContainerStarted","Data":"4b05b3620335ff600dcbaaba98c39ea5174b608163a35fbda19ee805218914c2"}
Nov 03 20:30:07.549989 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.549962    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ebc5d4efd8a63d505e3fa8dc27ffb1de","Type":"ContainerStarted","Data":"e47fd50ede4abe9baae7ae6cfe592c802481db097bf6ae16093d90460ecb983a"}
Nov 03 20:30:07.550105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.550065    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ba15cf602964f30d9df06d36baaae3a5","Type":"ContainerStarted","Data":"d821887e95a24f913bf9a83ba979e752664517672957fd30a19fa0bc19d14e2d"}
Nov 03 20:30:07.550105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.550099    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ba15cf602964f30d9df06d36baaae3a5","Type":"ContainerDied","Data":"796e1aebe4685889ab7430fa65c3a69062e94c71ee6502294df09a8eb67a4705"}
Nov 03 20:30:07.550219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.550121    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal" event={"ID":"ba15cf602964f30d9df06d36baaae3a5","Type":"ContainerStarted","Data":"c75fcf5bc1b7b5d205ec81a2ab8c1d8a480efd271e9ece8c6e27cbb118c8ccc9"}
Nov 03 20:30:07.550219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.550151    5318 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.550515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.550488    5318 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.559017 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.558997    5318 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.559090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.559049    5318 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.607755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.607734    5318 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:30:07.607830 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.607773    5318 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.608241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.608226    5318 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:30:07.608241 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:07.608264    5318 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.624727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.624702    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.624807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.624731    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.624807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.624748    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc5d4efd8a63d505e3fa8dc27ffb1de-config\") pod \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ebc5d4efd8a63d505e3fa8dc27ffb1de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.725648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.725627    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.725718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.725590    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.725718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.725701    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.725793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.725722    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc5d4efd8a63d505e3fa8dc27ffb1de-config\") pod \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ebc5d4efd8a63d505e3fa8dc27ffb1de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.725793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.725759    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc5d4efd8a63d505e3fa8dc27ffb1de-config\") pod \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ebc5d4efd8a63d505e3fa8dc27ffb1de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:07.725855 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:07.725789    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba15cf602964f30d9df06d36baaae3a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal\" (UID: \"ba15cf602964f30d9df06d36baaae3a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:08.125158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.125122    5318 apiserver.go:52] "Watching apiserver"
Nov 03 20:30:08.267832 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.267798    5318 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:30:08.268448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.268426    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k","openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l","openshift-multus/network-metrics-daemon-ms89f","openshift-monitoring/alertmanager-main-0","openshift-network-diagnostics/network-check-target-d82lk","openshift-insights/insights-runtime-extractor-vt4cx","openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6","openshift-ovn-kubernetes/ovnkube-node-j68vs","openshift-ingress-canary/ingress-canary-j6bcz","openshift-console/console-cb95cdfdd-vrl5j","openshift-multus/multus-knqnr","openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m","openshift-monitoring/metrics-server-674d9b67d4-m4jmd","openshift-image-registry/node-ca-kxpt4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-141.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-nl7nq","kube-system/konnectivity-agent-9k7ks","kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-7dw2d","openshift-cluster-node-tuning-operator/tuned-vmcgc","openshift-dns/node-resolver-hk7dd","openshift-monitoring/node-exporter-c4ksx","openshift-multus/multus-additional-cni-plugins-zpntw","openshift-dns/dns-default-5vdpt"]
Nov 03 20:30:08.316055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.316033    5318 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:08.317637 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.317620    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.317720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.317684    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:30:08.318454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318439    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.318454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318449    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.318557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318520    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.318557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318530    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.318557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318549    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.318664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318564    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.318700 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.318663    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.319625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.319598    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.319625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.319608    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:30:08.319737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.319714    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:30:08.319826 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.319808    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:30:08.319826 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.319810    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.319936 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.319925    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.320265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320250    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.320439 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320425    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.320516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320462    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.320623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320581    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:30:08.320623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320585    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:30:08.320623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320606    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:30:08.320879 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.320866    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.322321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.322306    5318 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:30:08.330100 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330084    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-socket-dir-parent\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.330146 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330111    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmww\" (UniqueName: \"kubernetes.io/projected/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-api-access-mcmww\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.330146 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330130    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtc8\" (UniqueName: \"kubernetes.io/projected/0762ba86-bec7-4282-a042-ca63ffaeeea9-kube-api-access-fmtc8\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.330238 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330145    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-etc-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.330310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330298    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-env-overrides\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.330362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330331    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-root\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.330493 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330476    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a641934f-d459-46af-be40-6aee398f3b82-host\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.330526 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330510    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4r2z\" (UniqueName: \"kubernetes.io/projected/0305c1b2-ad03-4e5b-847f-64b2828fedc2-kube-api-access-c4r2z\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.330584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330568    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.330620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330594    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-tls\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.330620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330612    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.330681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330636    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-host\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.330681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330665    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.330774 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330759    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cni-binary-copy\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.330806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330784    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovn-node-metrics-cert\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.330806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330800    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-kubelet-config\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.330861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330815    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:30:08.330904 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330889    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-etc-tuned\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.330934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330915    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.330963 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330935    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-node-log\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.330963 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330949    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331042 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330965    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-multus-certs\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331042 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330985    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-etc-tuned\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.331042 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.330984    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.331042 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331003    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cni-binary-copy\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331042 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331017    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.331199 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331090    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj82j\" (UniqueName: \"kubernetes.io/projected/bcba368e-649b-4dec-a6f3-edfb0f20ba38-kube-api-access-zj82j\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:08.331199 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331119    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-systemd\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.331199 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331141    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.331392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331240    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-multus\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331279    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-systemd-units\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.331392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331312    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-config\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.331392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331351    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-sys\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.331392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331367    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-conf-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331383    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.331598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331398    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.331598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331412    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.331598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331499    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.331598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331523    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.331598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331557    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlbz\" (UniqueName: \"kubernetes.io/projected/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-kube-api-access-jxlbz\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:30:08.331598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331578    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-tls\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.331779 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331715    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.331779 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331748    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-bin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331838 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331820    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-daemon-config\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.331868 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331849    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-data-volume\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.331915 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331868    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.331950 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331919    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.331980 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331959    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-data-volume\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.332040 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.331950    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.332089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332054    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-daemon-config\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.332089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332054    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchxh\" (UniqueName: \"kubernetes.io/projected/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-kube-api-access-hchxh\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.332185 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332116    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-textfile\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.332185 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332139    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcba368e-649b-4dec-a6f3-edfb0f20ba38-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:08.332185 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332154    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a641934f-d459-46af-be40-6aee398f3b82-serviceca\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.332185 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332176    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-netns\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.332368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332196    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-device-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.332368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332221    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rd6\" (UniqueName: \"kubernetes.io/projected/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-kube-api-access-48rd6\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.332368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332291    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-textfile\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.332368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332292    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-socket-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.332519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332378    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-kubelet\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.332519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332426    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmkw\" (UniqueName: \"kubernetes.io/projected/8cdac693-b083-4a91-bd5c-ea56d23bd771-kube-api-access-qzmkw\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.332519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332492    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.332674 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332524    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.332674 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332560    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgxb\" (UniqueName: \"kubernetes.io/projected/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-kube-api-access-6rgxb\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.332674 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332586    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7871e65-e639-4c12-ad40-4a2dd1b90687-iptables-alerter-script\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.332674 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332613    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-cnibin\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.332674 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332644    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cnibin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.332887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332685    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a641934f-d459-46af-be40-6aee398f3b82-serviceca\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.332887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332748    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7871e65-e639-4c12-ad40-4a2dd1b90687-iptables-alerter-script\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.332887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332748    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-netns\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.332887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332782    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-etc-kubernetes\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.332887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332840    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.332887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332861    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-sys\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.333092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.332962    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-log-socket\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.333092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333003    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12201904-e96b-4326-b87d-b5ac7539e37e-cert\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:30:08.333092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333084    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.333192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333111    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.333192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333140    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdbx\" (UniqueName: \"kubernetes.io/projected/42f1ee76-96ba-4983-b33a-b7321b678045-kube-api-access-6bdbx\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.333192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333155    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.333192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333177    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.333192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333171    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysconfig\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.333416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333304    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.333416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333353    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/056b04d8-72b3-4019-bddd-56a4469906de-tmp\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.333416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333414    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/056b04d8-72b3-4019-bddd-56a4469906de-tmp\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.333504 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333427    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-grpc-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.333504 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333453    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-hostroot\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.333504 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333471    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x82\" (UniqueName: \"kubernetes.io/projected/c7871e65-e639-4c12-ad40-4a2dd1b90687-kube-api-access-82x82\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.333582 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333560    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.333617 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333589    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-conf\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.333617 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333603    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-run\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.333690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333622    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-kubelet\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.333690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333666    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-slash\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.333756 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333731    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:30:08.333801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333757    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-hosts-file\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.333801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333784    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.333901 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333835    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wql\" (UniqueName: \"kubernetes.io/projected/17928b13-2c16-48bd-9cad-e8920d246891-kube-api-access-j2wql\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.333950 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333912    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0305c1b2-ad03-4e5b-847f-64b2828fedc2-metrics-tls\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.333999 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.333966    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-binary-copy\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.334039 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334003    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bce3daf-4de1-4fea-89e8-cd891e432355-konnectivity-ca\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:30:08.334077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334045    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.334111 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334073    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.334146 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334115    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.334175 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334161    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bce3daf-4de1-4fea-89e8-cd891e432355-konnectivity-ca\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:30:08.334213 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334185    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.334213 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334210    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.334310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334231    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.334310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334258    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.334310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334287    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-system-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.334415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334312    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-crio-socket\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.334415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334353    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-metrics-server-audit-profiles\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.334415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334381    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.334415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334410    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrv6d\" (UniqueName: \"kubernetes.io/projected/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-kube-api-access-rrv6d\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.334522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334433    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.334522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334440    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgf4\" (UniqueName: \"kubernetes.io/projected/2c38cada-a1d9-40a5-83c2-ac7157fb9975-kube-api-access-jqgf4\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.334522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334469    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17928b13-2c16-48bd-9cad-e8920d246891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.334522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334474    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.334628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334519    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.334680 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334664    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-k8s-cni-cncf-io\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.334777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334763    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.334832 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334794    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-wtmp\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.334832 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334819    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.334923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334838    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-client-certs\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.334923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334871    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bce3daf-4de1-4fea-89e8-cd891e432355-agent-certs\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:30:08.334923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334916    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.335074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334945    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-sys-fs\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.335074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334973    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0762ba86-bec7-4282-a042-ca63ffaeeea9-audit-log\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.335074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.334998    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-client-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.335074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335026    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-var-lib-kubelet\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.335074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335056    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0305c1b2-ad03-4e5b-847f-64b2828fedc2-config-volume\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335076    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0762ba86-bec7-4282-a042-ca63ffaeeea9-audit-log\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335083    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0305c1b2-ad03-4e5b-847f-64b2828fedc2-tmp-dir\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335148    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-systemd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335180    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0305c1b2-ad03-4e5b-847f-64b2828fedc2-tmp-dir\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335187    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-script-lib\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335215    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-tmp-dir\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.335241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335239    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335258    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-kubernetes\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335279    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-registration-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335293    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-tmp-dir\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335308    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbtk\" (UniqueName: \"kubernetes.io/projected/19efb517-1936-48f3-ab42-bd7ee7a87239-kube-api-access-tpbtk\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335351    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-tmp\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335461    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f1ee76-96ba-4983-b33a-b7321b678045-tmp\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.335503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335486    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335512    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335529    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335546    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-netd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335585    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056b04d8-72b3-4019-bddd-56a4469906de-metrics-client-ca\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335637    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh977\" (UniqueName: \"kubernetes.io/projected/056b04d8-72b3-4019-bddd-56a4469906de-kube-api-access-gh977\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335656    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-metrics-client-ca\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335670    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335694    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-system-cni-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.335749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335717    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjr6x\" (UniqueName: \"kubernetes.io/projected/a641934f-d459-46af-be40-6aee398f3b82-kube-api-access-vjr6x\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335767    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335814    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335838    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7871e65-e639-4c12-ad40-4a2dd1b90687-host-slash\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335863    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-modprobe-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335935    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-ovn\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.335971    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336000    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfnj\" (UniqueName: \"kubernetes.io/projected/12201904-e96b-4326-b87d-b5ac7539e37e-kube-api-access-lxfnj\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336040    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-var-lib-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336058    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-bin\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336074    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.336106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336097    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.336441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336160    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.336441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336211    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-dbus\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.336441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.336228    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-lib-modules\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.349980 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.349955    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:30:08.350060 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.349988    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bb4f5\""
Nov 03 20:30:08.350060 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350042    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7m98\""
Nov 03 20:30:08.350192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350179    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:30:08.350395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350377    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\""
Nov 03 20:30:08.350455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350402    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dfjfq\""
Nov 03 20:30:08.350455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350419    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\""
Nov 03 20:30:08.350455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350418    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:30:08.350455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350432    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qtf8\""
Nov 03 20:30:08.350455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350449    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8tnq\""
Nov 03 20:30:08.350628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350465    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\""
Nov 03 20:30:08.350628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350467    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kkpg\""
Nov 03 20:30:08.350628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350487    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:30:08.350628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350526    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7b8b\""
Nov 03 20:30:08.350628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.350587    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ff7gv\""
Nov 03 20:30:08.351502 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.351486    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-metrics-certs\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:30:08.353864 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.353847    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7574d083-15a3-4cf9-9547-93c53eb8222a-original-pull-secret\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.356077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.356062    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bce3daf-4de1-4fea-89e8-cd891e432355-agent-certs\") pod \"konnectivity-agent-9k7ks\" (UID: \"4bce3daf-4de1-4fea-89e8-cd891e432355\") " pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:30:08.363819 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.363798    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:30:08.363819 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.363816    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:30:08.364278 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364256    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.364350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364296    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.364350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364296    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:30:08.364449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364378    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:30:08.364550 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364533    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:30:08.364550 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364544    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.364631 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364575    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.364631 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364583    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.364631 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364604    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:30:08.364726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364642    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.364726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364680    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:30:08.364726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364688    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.364726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364700    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:30:08.364830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364779    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:30:08.364830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.364786    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:30:08.365327 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.365313    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0305c1b2-ad03-4e5b-847f-64b2828fedc2-config-volume\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.365370 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.365331    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:30:08.365469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.365456    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-script-lib\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.366593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.366572    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.366666 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.366652    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/056b04d8-72b3-4019-bddd-56a4469906de-metrics-client-ca\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.366706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.366656    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-metrics-client-ca\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.366706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.366685    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.370879 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.370865    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-env-overrides\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.371610 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.371596    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovnkube-config\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.372173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.372160    5318 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:30:08.375592 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.372414    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.375592 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:08.372489    5318 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal"
Nov 03 20:30:08.375592 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.372964    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.376030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.376010    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.376030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.376019    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.376119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.376019    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.389532 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.389512    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdbx\" (UniqueName: \"kubernetes.io/projected/42f1ee76-96ba-4983-b33a-b7321b678045-kube-api-access-6bdbx\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.391779 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.391763    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4r2z\" (UniqueName: \"kubernetes.io/projected/0305c1b2-ad03-4e5b-847f-64b2828fedc2-kube-api-access-c4r2z\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.391911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.391891    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rd6\" (UniqueName: \"kubernetes.io/projected/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-kube-api-access-48rd6\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.392022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.392007    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmkw\" (UniqueName: \"kubernetes.io/projected/8cdac693-b083-4a91-bd5c-ea56d23bd771-kube-api-access-qzmkw\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.392121 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.392101    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cph\" (UniqueName: \"kubernetes.io/projected/55f1e1e3-49b2-4665-b2a7-5e0e807d0cee-kube-api-access-s8cph\") pod \"network-check-target-d82lk\" (UID: \"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee\") " pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:30:08.392161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.392118    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrv6d\" (UniqueName: \"kubernetes.io/projected/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-kube-api-access-rrv6d\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.394643 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.394627    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wql\" (UniqueName: \"kubernetes.io/projected/17928b13-2c16-48bd-9cad-e8920d246891-kube-api-access-j2wql\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.395184 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.395165    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgxb\" (UniqueName: \"kubernetes.io/projected/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-kube-api-access-6rgxb\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.395218 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.395191    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmww\" (UniqueName: \"kubernetes.io/projected/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-kube-api-access-mcmww\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.395261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.395247    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x82\" (UniqueName: \"kubernetes.io/projected/c7871e65-e639-4c12-ad40-4a2dd1b90687-kube-api-access-82x82\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.395424 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.395411    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj82j\" (UniqueName: \"kubernetes.io/projected/bcba368e-649b-4dec-a6f3-edfb0f20ba38-kube-api-access-zj82j\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:08.396823 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.396808    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.397262 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.397241    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchxh\" (UniqueName: \"kubernetes.io/projected/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-kube-api-access-hchxh\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.397262 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.397252    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlbz\" (UniqueName: \"kubernetes.io/projected/9285d5c2-ae97-46ad-974a-2e2cbce9b36e-kube-api-access-jxlbz\") pod \"network-metrics-daemon-ms89f\" (UID: \"9285d5c2-ae97-46ad-974a-2e2cbce9b36e\") " pod="openshift-multus/network-metrics-daemon-ms89f"
Nov 03 20:30:08.397496 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.397481    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtc8\" (UniqueName: \"kubernetes.io/projected/0762ba86-bec7-4282-a042-ca63ffaeeea9-kube-api-access-fmtc8\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.406033 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.406015    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfnj\" (UniqueName: \"kubernetes.io/projected/12201904-e96b-4326-b87d-b5ac7539e37e-kube-api-access-lxfnj\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:30:08.406359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.406325    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.411530 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.411509    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbtk\" (UniqueName: \"kubernetes.io/projected/19efb517-1936-48f3-ab42-bd7ee7a87239-kube-api-access-tpbtk\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.411596 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.411544    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjr6x\" (UniqueName: \"kubernetes.io/projected/a641934f-d459-46af-be40-6aee398f3b82-kube-api-access-vjr6x\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.411764 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.411749    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgf4\" (UniqueName: \"kubernetes.io/projected/2c38cada-a1d9-40a5-83c2-ac7157fb9975-kube-api-access-jqgf4\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.411829 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.411813    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh977\" (UniqueName: \"kubernetes.io/projected/056b04d8-72b3-4019-bddd-56a4469906de-kube-api-access-gh977\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.425274 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425251    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 20:30:08.425376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425269    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:30:08.425376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425324    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:30:08.425516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425500    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 20:30:08.425580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425502    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:30:08.425580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425502    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:30:08.425580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425538    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:30:08.425697 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425593    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:30:08.425697 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425636    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:30:08.425769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425708    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 20:30:08.425798 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425785    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.425906 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425892    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:30:08.425906 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425901    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 20:30:08.426013 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.425996    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9t9mk\""
Nov 03 20:30:08.426165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426049    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdt7f\""
Nov 03 20:30:08.426165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426134    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w44xl\""
Nov 03 20:30:08.426290 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426239    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-en5e1gdr1rmv0\""
Nov 03 20:30:08.426290 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426273    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g5f8g\""
Nov 03 20:30:08.426389 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426308    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tmzf6\""
Nov 03 20:30:08.426431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426417    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:30:08.426508 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426492    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hl5hz\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426518    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2str7q7cdaj8\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426526    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426563    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426572    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426587    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426587    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:30:08.426669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426613    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:30:08.426894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426758    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:30:08.426894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426791    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 20:30:08.426894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.426877    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4nsd9\""
Nov 03 20:30:08.427094 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.427081    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nczmv\""
Nov 03 20:30:08.427510 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.427499    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 20:30:08.428517 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428506    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 20:30:08.428554 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428529    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 20:30:08.428619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428607    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:30:08.428778 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428765    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 20:30:08.428778 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428770    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:30:08.428865 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428796    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c6v9\""
Nov 03 20:30:08.428901 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428878    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9nsh4\""
Nov 03 20:30:08.428901 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428892    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:30:08.428965 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.428914    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qfcmp\""
Nov 03 20:30:08.429661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.429648    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jhfbw\""
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431891    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431906    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431922    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-tls\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431936    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431943    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431925    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431925    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.431974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.431922    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cdac693-b083-4a91-bd5c-ea56d23bd771-ovn-node-metrics-cert\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.432361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.432043    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.432361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.432047    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-tls\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.432361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.432071    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/298b5083-4edb-4714-96e4-a9d9a9a4a1e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jgv9m\" (UID: \"298b5083-4edb-4714-96e4-a9d9a9a4a1e1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jgv9m"
Nov 03 20:30:08.432361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.432159    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.432361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.432202    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.433239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.433219    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12201904-e96b-4326-b87d-b5ac7539e37e-cert\") pod \"ingress-canary-j6bcz\" (UID: \"12201904-e96b-4326-b87d-b5ac7539e37e\") " pod="openshift-ingress-canary/ingress-canary-j6bcz"
Nov 03 20:30:08.433289 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.433239    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cae6ab2-e17e-4b19-8e99-f36f7072a3b7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x8c4k\" (UID: \"3cae6ab2-e17e-4b19-8e99-f36f7072a3b7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x8c4k"
Nov 03 20:30:08.433289 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.433256    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bcba368e-649b-4dec-a6f3-edfb0f20ba38-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-9gkw6\" (UID: \"bcba368e-649b-4dec-a6f3-edfb0f20ba38\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:08.433289 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.433264    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.434443 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434423    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.434521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434478    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.434521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434495    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0305c1b2-ad03-4e5b-847f-64b2828fedc2-metrics-tls\") pod \"dns-default-5vdpt\" (UID: \"0305c1b2-ad03-4e5b-847f-64b2828fedc2\") " pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:08.434644 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434566    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.434644 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434573    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.434644 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434633    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-grpc-tls\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.434809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434794    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.434850 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.434823    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/056b04d8-72b3-4019-bddd-56a4469906de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85d9cd56d8-tsp6l\" (UID: \"056b04d8-72b3-4019-bddd-56a4469906de\") " pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:08.435750 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.435735    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-secret-metrics-server-client-certs\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.435809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.435774    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.435809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.435778    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0762ba86-bec7-4282-a042-ca63ffaeeea9-client-ca-bundle\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.436884 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.436861    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.436997 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.436981    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysconfig\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437013    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-hostroot\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437039    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-conf\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437061    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysconfig\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437062    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-run\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437093    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-kubelet\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437107    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-run\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437113    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-slash\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437115    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-hostroot\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437139    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-kubelet\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437151    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-hosts-file\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437154    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-slash\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437181    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-system-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437206    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-crio-socket\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437228    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0be5ee45-4c57-4f1f-a3a1-075c2dca3c90-hosts-file\") pod \"node-resolver-hk7dd\" (UID: \"0be5ee45-4c57-4f1f-a3a1-075c2dca3c90\") " pod="openshift-dns/node-resolver-hk7dd"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437207    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-conf\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437240    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-system-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437232    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437271    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48361014-a0f8-4050-8fdf-bf7d7b70ddcb-crio-socket\") pod \"insights-runtime-extractor-vt4cx\" (UID: \"48361014-a0f8-4050-8fdf-bf7d7b70ddcb\") " pod="openshift-insights/insights-runtime-extractor-vt4cx"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437274    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-k8s-cni-cncf-io\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437300    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437322    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-k8s-cni-cncf-io\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437323    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-wtmp\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437363    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437365    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-sysctl-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437363    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-sys-fs\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437394    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-var-lib-kubelet\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437415    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-systemd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437405    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-wtmp\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437414    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-sys-fs\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437447    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-systemd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437444    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-var-lib-kubelet\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437464    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-kubernetes\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.437472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437480    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-registration-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437512    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-registration-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437524    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-kubernetes\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437542    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-netd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437563    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-system-cni-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437579    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7871e65-e639-4c12-ad40-4a2dd1b90687-host-slash\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437589    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-netd\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437594    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-modprobe-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437609    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-ovn\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437625    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-var-lib-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437629    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-system-cni-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437640    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-bin\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437650    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7871e65-e639-4c12-ad40-4a2dd1b90687-host-slash\") pod \"iptables-alerter-7dw2d\" (UID: \"c7871e65-e639-4c12-ad40-4a2dd1b90687\") " pod="openshift-network-operator/iptables-alerter-7dw2d"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437672    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437682    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-run-ovn\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437688    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-dbus\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437696    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-modprobe-d\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437710    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-cni-bin\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437721    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-lib-modules\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437722    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-dbus\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437738    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-var-lib-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437779    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437800    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-socket-dir-parent\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437863    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-etc-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437883    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-root\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437886    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-lib-modules\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437887    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-socket-dir-parent\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437908    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-etc-openvswitch\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437915    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-root\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437922    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a641934f-d459-46af-be40-6aee398f3b82-host\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437951    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-host\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437954    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a641934f-d459-46af-be40-6aee398f3b82-host\") pod \"node-ca-kxpt4\" (UID: \"a641934f-d459-46af-be40-6aee398f3b82\") " pod="openshift-image-registry/node-ca-kxpt4"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.437996    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-kubelet-config\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438013    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-host\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438017    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-node-log\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438031    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438038    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7574d083-15a3-4cf9-9547-93c53eb8222a-kubelet-config\") pod \"global-pull-secret-syncer-nl7nq\" (UID: \"7574d083-15a3-4cf9-9547-93c53eb8222a\") " pod="kube-system/global-pull-secret-syncer-nl7nq"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438046    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-multus-certs\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438069    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438084    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-node-log\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438092    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438109    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-cni-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438116    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-systemd\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438126    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438125    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-multus-certs\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438141    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438139    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-multus\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438164    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-systemd-units\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438169    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-multus\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438178    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-sys\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438196    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-sys\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438174    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-etc-systemd\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438212    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-conf-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438225    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-multus-conf-dir\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438231    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-systemd-units\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438259    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438290    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438294    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438313    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-ovn-kubernetes\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.438317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438324    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-bin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438372    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-var-lib-cni-bin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438378    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-netns\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438400    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-device-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438416    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-socket-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438431    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-kubelet\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438454    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-device-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438455    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-cnibin\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438477    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-run-netns\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438477    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cnibin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438501    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-host-kubelet\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438498    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19efb517-1936-48f3-ab42-bd7ee7a87239-socket-dir\") pod \"aws-ebs-csi-driver-node-hqprf\" (UID: \"19efb517-1936-48f3-ab42-bd7ee7a87239\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqprf"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438519    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-netns\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438537    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-host-run-netns\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438521    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-cnibin\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438539    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17928b13-2c16-48bd-9cad-e8920d246891-cnibin\") pod \"multus-additional-cni-plugins-zpntw\" (UID: \"17928b13-2c16-48bd-9cad-e8920d246891\") " pod="openshift-multus/multus-additional-cni-plugins-zpntw"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438568    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-etc-kubernetes\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438594    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-sys\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438611    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c38cada-a1d9-40a5-83c2-ac7157fb9975-etc-kubernetes\") pod \"multus-knqnr\" (UID: \"2c38cada-a1d9-40a5-83c2-ac7157fb9975\") " pod="openshift-multus/multus-knqnr"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438616    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-log-socket\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438666    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42f1ee76-96ba-4983-b33a-b7321b678045-sys\") pod \"tuned-vmcgc\" (UID: \"42f1ee76-96ba-4983-b33a-b7321b678045\") " pod="openshift-cluster-node-tuning-operator/tuned-vmcgc"
Nov 03 20:30:08.439991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.438676    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8cdac693-b083-4a91-bd5c-ea56d23bd771-log-socket\") pod \"ovnkube-node-j68vs\" (UID: \"8cdac693-b083-4a91-bd5c-ea56d23bd771\") " pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:08.462746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.462725    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:30:08.462746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.462728    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 20:30:08.462857 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.462759    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:30:08.463842 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.463826    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:30:08.464881 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.464865    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") pod \"console-cb95cdfdd-vrl5j\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") " pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:08.464982 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.464967    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0762ba86-bec7-4282-a042-ca63ffaeeea9-metrics-server-audit-profiles\") pod \"metrics-server-674d9b67d4-m4jmd\" (UID: \"0762ba86-bec7-4282-a042-ca63ffaeeea9\") " pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:08.465120 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.465106    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1af07fe5-98e6-4f65-a70f-1b5cfe9c1735-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4ksx\" (UID: \"1af07fe5-98e6-4f65-a70f-1b5cfe9c1735\") " pod="openshift-monitoring/node-exporter-c4ksx"
Nov 03 20:30:08.465225 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.465210    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:08.571782 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:08.571760    5318 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:30:09.191147 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.191117    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:09.389183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.389141    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:09.399669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.399649    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5vdpt"
Nov 03 20:30:09.781508 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.781477    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:09.828547 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.828516    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:09.855707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.855675    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.870707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.870683    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:09.900809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.900784    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:09.908074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.908037    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:30:09.908497 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.908463    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:30:09.909001 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.908958    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:30:09.910684 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.910407    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:30:09.915367 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.911975    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:30:09.949449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949421    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949466    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949496    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949597    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949682    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949740    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949921 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949787    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949921 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949820    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949921 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949836    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.949921 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949887    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.949958    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950004    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950039    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950062    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950085    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlz5z\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950109    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950140    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950163    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950199    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.950353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.950256    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965699    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965732    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965757    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965778    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965807    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965819    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965843    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w7ljs\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965863    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:30:09.965945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.965914    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1gjsachoofh47\""
Nov 03 20:30:09.967047 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.966231    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:30:09.967047 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:09.966907    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:30:10.050811 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050789    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.050911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050826    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.050911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050853    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.050911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050871    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.050911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050887    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.050911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050907    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050929    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050945    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050961    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.050975    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051002    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051022    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051036    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051051    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlz5z\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051075    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051102    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051118    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051141    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051164    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051183    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.051699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.051512    5318 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:30:10.052646 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.052189    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.053389 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.053368    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.053572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.053549    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.055557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.054066    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.055557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.054663    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.055557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.055310    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.056626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.056600    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.056732 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.056667    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.056841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.056818    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.057452 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.057136    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.057452 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.057268    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.058809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.057926    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.058809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.057994    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.058809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.058012    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.058809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.058387    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.058809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.058761    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.059597 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.058832    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.059597 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.058864    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.059917 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.059895    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.115731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.115712    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlz5z\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z\") pod \"prometheus-k8s-0\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.326414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.326330    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:10.330273 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.330246    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d82lk" event={"ID":"55f1e1e3-49b2-4665-b2a7-5e0e807d0cee","Type":"ContainerStarted","Data":"10690ee7785e9bdbc1830f6f67b7235756638225dd64518af92ab680b0882eca"}
Nov 03 20:30:10.333647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.333613    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"7533fd66b4f1e2aacbf1ff08e5042a31e9b8f9d31523ccfe36c9253e5e1243fd"}
Nov 03 20:30:10.333731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.333682    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"b528696c6ab6a0e98b79594ff8e14ea1feb943de343c622d7a70a021a34b1011"}
Nov 03 20:30:10.333792 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.333739    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"c9697c8969e65ba2fe9dd4476d1fba007baf4e6256fe6b14362dc0b7a16916a7"}
Nov 03 20:30:10.775421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:10.775392    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:10.779725 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:10.779667    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7beedd_4ffd_432f_9a23_206aac61c107.slice/crio-2f093cc341779b078e8a5979e9282247ac90a263a94e0d492019f27c46edd70b WatchSource:0}: Error finding container 2f093cc341779b078e8a5979e9282247ac90a263a94e0d492019f27c46edd70b: Status 404 returned error can't find the container with id 2f093cc341779b078e8a5979e9282247ac90a263a94e0d492019f27c46edd70b
Nov 03 20:30:11.179000 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.178971    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c584b584d-v26mn"]
Nov 03 20:30:11.194618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.194584    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.244485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.244459    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c584b584d-v26mn"]
Nov 03 20:30:11.251064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.251038    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:30:11.263173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263153    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263189    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263219    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdnw\" (UniqueName: \"kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263384    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263437 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263418    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263495 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263481    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263532 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263522    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.263575 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.263543    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.342102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.342059    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a" exitCode=0
Nov 03 20:30:11.342102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.342094    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"}
Nov 03 20:30:11.342516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.342165    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"2f093cc341779b078e8a5979e9282247ac90a263a94e0d492019f27c46edd70b"}
Nov 03 20:30:11.364417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.364393    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.364660 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.364642    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.364741 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.364686    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.364796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.364747    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.364848 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.364839    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.365010 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.364971    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.365086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.365028    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.365543 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.365524    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.365626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.365575    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.365626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.365596    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdnw\" (UniqueName: \"kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.365720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.365677    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.366080 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.366058    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.366244 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.366225    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.367305 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.367286    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.367740 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.367719    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.423037 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.423006    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdnw\" (UniqueName: \"kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw\") pod \"console-7c584b584d-v26mn\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") " pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.451136 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.451111    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:30:11.525681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.525657    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:11.904324 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:11.904297    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c584b584d-v26mn"]
Nov 03 20:30:11.908237 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:11.908207    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917ad3c9_eea6_4f27_b366_26fd14c8ab0d.slice/crio-363e071b50d76c5306d5fbe7977129abcaa69703fa0fdb948c81bccc9aa774c3 WatchSource:0}: Error finding container 363e071b50d76c5306d5fbe7977129abcaa69703fa0fdb948c81bccc9aa774c3: Status 404 returned error can't find the container with id 363e071b50d76c5306d5fbe7977129abcaa69703fa0fdb948c81bccc9aa774c3
Nov 03 20:30:12.347939 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.347905    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd" event={"ID":"0762ba86-bec7-4282-a042-ca63ffaeeea9","Type":"ContainerStarted","Data":"5247a988617453c3e789eea5201c96522835799dc6279c2fd467903c22d168be"}
Nov 03 20:30:12.349404 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.349376    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6" event={"ID":"bcba368e-649b-4dec-a6f3-edfb0f20ba38","Type":"ContainerStarted","Data":"4772ae07e2999db632a97ade0df6a21f738bf968098ca417649b4aa5e9a176b4"}
Nov 03 20:30:12.350791 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.350766    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584b584d-v26mn" event={"ID":"917ad3c9-eea6-4f27-b366-26fd14c8ab0d","Type":"ContainerStarted","Data":"71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf"}
Nov 03 20:30:12.350880 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.350810    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584b584d-v26mn" event={"ID":"917ad3c9-eea6-4f27-b366-26fd14c8ab0d","Type":"ContainerStarted","Data":"363e071b50d76c5306d5fbe7977129abcaa69703fa0fdb948c81bccc9aa774c3"}
Nov 03 20:30:12.513185 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:12.513130    5318 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage2700426860/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/28/28ca5925cb2de12257139d8e97c4a489caf6d6967250aef2d50ce58edf878667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T203011Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4a87e1e03dcb54578af218ad032f163e6098643fa93aadbaad6ade94bb9f7b12&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793738711~hmac=74b68b6032f009e56814553fa2c5f9c4818dc0a40add2c96d352d4f848392ae8\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:de5f95977e32d65d7f0586eb297cd098da0acaaffb3512483d50bd6eda40bfb8"
Nov 03 20:30:12.513481 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:12.513441    5318 kuberuntime_manager.go:1707] "Unhandled Error" err="container alertmanager start failed in pod alertmanager-main-0_openshift-monitoring(9459b28f-1cb6-4b5f-b0df-f30ccd9f5470): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage2700426860/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/28/28ca5925cb2de12257139d8e97c4a489caf6d6967250aef2d50ce58edf878667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T203011Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4a87e1e03dcb54578af218ad032f163e6098643fa93aadbaad6ade94bb9f7b12&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793738711~hmac=74b68b6032f009e56814553fa2c5f9c4818dc0a40add2c96d352d4f848392ae8\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError"
Nov 03 20:30:12.663816 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.663730    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:12.669668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.669634    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-9gkw6"
Nov 03 20:30:12.682149 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.682099    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c584b584d-v26mn" podStartSLOduration=1.682083019 podStartE2EDuration="1.682083019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:12.681107165 +0000 UTC m=+5.660664007" watchObservedRunningTime="2026-11-03 20:30:12.682083019 +0000 UTC m=+5.661639860"
Nov 03 20:30:12.907847 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:12.907825    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:13.142616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.142585    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:13.253555 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:13.253513    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"alertmanager\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2700426860/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/28/28ca5925cb2de12257139d8e97c4a489caf6d6967250aef2d50ce58edf878667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T203011Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4a87e1e03dcb54578af218ad032f163e6098643fa93aadbaad6ade94bb9f7b12&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793738711~hmac=74b68b6032f009e56814553fa2c5f9c4818dc0a40add2c96d352d4f848392ae8\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"
Nov 03 20:30:13.358404 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.358369    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"b025227f6c6942f77a1eb2391f29d4bef9dae4f0ab8a07f93fdbd9a3af9f03f2"}
Nov 03 20:30:13.358845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.358435    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"a5bffd8ca2297cf961279314ef3545bb3131b8c126f90c8cfd50db7908f37790"}
Nov 03 20:30:13.358845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.358472    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l" event={"ID":"056b04d8-72b3-4019-bddd-56a4469906de","Type":"ContainerStarted","Data":"37059d16b2782bc50318026eb1c59754ca1544327681e8a5639b1f94c716fdee"}
Nov 03 20:30:13.358845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.358697    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:13.362534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.362394    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4"}
Nov 03 20:30:13.362534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.362477    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26"}
Nov 03 20:30:13.362534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.362507    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3"}
Nov 03 20:30:13.362534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.362531    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb"}
Nov 03 20:30:13.362796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:13.362563    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238"}
Nov 03 20:30:13.364488 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:13.364452    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"alertmanager\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:de5f95977e32d65d7f0586eb297cd098da0acaaffb3512483d50bd6eda40bfb8\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2700426860/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/28/28ca5925cb2de12257139d8e97c4a489caf6d6967250aef2d50ce58edf878667?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261103T203011Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4a87e1e03dcb54578af218ad032f163e6098643fa93aadbaad6ade94bb9f7b12&region=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793738711~hmac=74b68b6032f009e56814553fa2c5f9c4818dc0a40add2c96d352d4f848392ae8\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"
Nov 03 20:30:14.415146 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:14.415116    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:14.722049 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:14.721946    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:14.922419 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:14.922366    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:15.118035 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.118012    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:15.376505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.376471    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"}
Nov 03 20:30:15.376653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.376535    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"}
Nov 03 20:30:15.376653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.376561    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"}
Nov 03 20:30:15.376653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.376584    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"}
Nov 03 20:30:15.376653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.376610    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"}
Nov 03 20:30:15.376653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.376632    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerStarted","Data":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"}
Nov 03 20:30:15.542123 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.541991    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.92737466 podStartE2EDuration="6.541976744s" totalImagesPullingTime="3.595713692s" totalInitContainerRuntime="18.888392ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:15.540988015 +0000 UTC m=+8.520544856" watchObservedRunningTime="2026-11-03 20:30:15.541976744 +0000 UTC m=+8.521533581"
Nov 03 20:30:15.873897 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:15.873863    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:16.078101 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:16.078070    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:17.343171 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:17.343139    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:17.967317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:17.967286    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:17.967589 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:17.967571    5318 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:30:18.486599 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:18.486567    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9k7ks"
Nov 03 20:30:18.519645 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:18.519622    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:18.852426 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:18.852392    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:18.871249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:18.871226    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:19.170277 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:19.170192    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:19.175124 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:19.174907    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:19.371796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:19.371771    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85d9cd56d8-tsp6l"
Nov 03 20:30:19.394103 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:19.394079    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:30:19.405220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:19.405203    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:19.762826 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:19.762794    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb95cdfdd-vrl5j"]
Nov 03 20:30:20.264256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:20.264218    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:25.241216 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:25.241181    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:25.440152 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:25.440123    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:25.932687 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:25.932654    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:30:26.854353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:26.854311    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:29.943223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:29.943187    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:30.146108 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:30.146075    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:31.438861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:31.438825    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerStarted","Data":"9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e"}
Nov 03 20:30:32.429689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:32.429651    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:32.628579 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:32.628540    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:34.744626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:34.744598    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j68vs"
Nov 03 20:30:40.269954 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:40.269919    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:40.273632 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:40.273610    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-674d9b67d4-m4jmd"
Nov 03 20:30:41.454375 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:41.454327    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d82lk"
Nov 03 20:30:44.786840 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:44.786785    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-cb95cdfdd-vrl5j" podUID="336018cd-e7cc-4f23-809a-8b67aebd020d" containerName="console" containerID="cri-o://b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421" gracePeriod=15
Nov 03 20:30:45.031814 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.031787    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:45.066583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066515    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066555    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066593    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066617    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066654    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066736    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066971 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066778    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") pod \"336018cd-e7cc-4f23-809a-8b67aebd020d\" (UID: \"336018cd-e7cc-4f23-809a-8b67aebd020d\") "
Nov 03 20:30:45.066971 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.066953    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:45.067080 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.067005    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:45.067204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.067164    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:45.067326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.067198    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:45.067326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.067180    5318 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.067326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.067237    5318 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-oauth-serving-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.068912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.068888    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:45.069286 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.069263    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:45.069396 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.069283    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw" pod "336018cd-e7cc-4f23-809a-8b67aebd020d" (UID: "336018cd-e7cc-4f23-809a-8b67aebd020d"). InnerVolumeSpecName "kube-api-access-b95rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:45.167627 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.167601    5318 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-oauth-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.167627 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.167624    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95rw\" (UniqueName: \"kubernetes.io/projected/336018cd-e7cc-4f23-809a-8b67aebd020d-kube-api-access-b95rw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.167775 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.167637    5318 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/336018cd-e7cc-4f23-809a-8b67aebd020d-console-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.167775 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.167647    5318 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/336018cd-e7cc-4f23-809a-8b67aebd020d-tmp\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.167775 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.167657    5318 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/336018cd-e7cc-4f23-809a-8b67aebd020d-console-serving-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:45.489524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.489488    5318 generic.go:356] "Generic (PLEG): container finished" podID="336018cd-e7cc-4f23-809a-8b67aebd020d" containerID="b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421" exitCode=0
Nov 03 20:30:45.489709 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.489545    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb95cdfdd-vrl5j"
Nov 03 20:30:45.489709 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.489548    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb95cdfdd-vrl5j" event={"ID":"336018cd-e7cc-4f23-809a-8b67aebd020d","Type":"ContainerDied","Data":"b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421"}
Nov 03 20:30:45.489709 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.489661    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb95cdfdd-vrl5j" event={"ID":"336018cd-e7cc-4f23-809a-8b67aebd020d","Type":"ContainerDied","Data":"9eaa0d8f93e0ce66a943599bff38dc1d4b713c0d8af20de4b19331c279f391f2"}
Nov 03 20:30:45.489709 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.489692    5318 scope.go:122] "RemoveContainer" containerID="b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421"
Nov 03 20:30:45.498753 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.498733    5318 scope.go:122] "RemoveContainer" containerID="b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421"
Nov 03 20:30:45.499009 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:45.498992    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421\": container with ID starting with b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421 not found: ID does not exist" containerID="b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421"
Nov 03 20:30:45.499058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.499017    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421"} err="failed to get container status \"b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421\": rpc error: code = NotFound desc = could not find container \"b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421\": container with ID starting with b6c04791bb8ba1879d0e03e238a6538df06c287c55d48bf587356969cef03421 not found: ID does not exist"
Nov 03 20:30:45.641899 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.641876    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb95cdfdd-vrl5j"]
Nov 03 20:30:45.688252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:45.688232    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cb95cdfdd-vrl5j"]
Nov 03 20:30:46.356163 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:46.356130    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:46.605854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:46.605822    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:47.252760 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:47.252727    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336018cd-e7cc-4f23-809a-8b67aebd020d" path="/var/lib/kubelet/pods/336018cd-e7cc-4f23-809a-8b67aebd020d/volumes"
Nov 03 20:30:48.586690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.586655    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:48.587189 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.586991    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="config-reloader" containerID="cri-o://c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238" gracePeriod=120
Nov 03 20:30:48.587189 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.587057    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="prom-label-proxy" containerID="cri-o://2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4" gracePeriod=120
Nov 03 20:30:48.587189 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.587093    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy" containerID="cri-o://d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3" gracePeriod=120
Nov 03 20:30:48.587189 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.587113    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="alertmanager" containerID="cri-o://9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e" gracePeriod=120
Nov 03 20:30:48.587456 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.587210    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-metric" containerID="cri-o://eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26" gracePeriod=120
Nov 03 20:30:48.587456 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:48.587206    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-web" containerID="cri-o://f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb" gracePeriod=120
Nov 03 20:30:49.508535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508503    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e" exitCode=0
Nov 03 20:30:49.508535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508529    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4" exitCode=0
Nov 03 20:30:49.508535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508536    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26" exitCode=0
Nov 03 20:30:49.508535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508541    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3" exitCode=0
Nov 03 20:30:49.508794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508550    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238" exitCode=0
Nov 03 20:30:49.508794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508572    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e"}
Nov 03 20:30:49.508794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508629    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4"}
Nov 03 20:30:49.508794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508657    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26"}
Nov 03 20:30:49.508794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508680    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3"}
Nov 03 20:30:49.508794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.508702    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238"}
Nov 03 20:30:49.838101 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.838076    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:49.905355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905310    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905373    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905413    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905469    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905641    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905795 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905676    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:49.905795 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905703    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905795 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905735    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.905795 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905782    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.906031 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905810    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.906031 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905853    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.906031 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905881    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.906031 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905911    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.906031 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.905953    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") pod \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\" (UID: \"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470\") "
Nov 03 20:30:49.906295 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.906101    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:49.906295 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.906279    5318 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-main-db\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:49.906422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.906301    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-metrics-client-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:49.907240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.907199    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:49.909362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909310    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:49.909502 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909481    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "kube-api-access-g4lqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:49.909644 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909621    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:49.909738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909668    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:49.909738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909695    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:49.909851 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909791    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:49.909888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.909852    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:49.910172 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.910149    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:49.913141 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.913098    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:49.919134 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:49.919111    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config" pod "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" (UID: "9459b28f-1cb6-4b5f-b0df-f30ccd9f5470"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:50.006846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006822    5318 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-cluster-tls-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006845    5318 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006856    5318 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-web-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006866    5318 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-main-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006875    5318 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006885    5318 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-config-out\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006894    5318 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006903    5318 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006912    5318 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006921    5318 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-tls-assets\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.006992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.006929    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lqq\" (UniqueName: \"kubernetes.io/projected/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470-kube-api-access-g4lqq\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:50.515188 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.515154    5318 generic.go:356] "Generic (PLEG): container finished" podID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerID="f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb" exitCode=0
Nov 03 20:30:50.515365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.515210    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb"}
Nov 03 20:30:50.515365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.515230    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.515365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.515268    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9459b28f-1cb6-4b5f-b0df-f30ccd9f5470","Type":"ContainerDied","Data":"b534ead6333a7acae1bac241d2c3eccb036f2c5b9e1cb81d6a84202ec61ccd68"}
Nov 03 20:30:50.515365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.515313    5318 scope.go:122] "RemoveContainer" containerID="9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e"
Nov 03 20:30:50.524301 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.524280    5318 scope.go:122] "RemoveContainer" containerID="2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4"
Nov 03 20:30:50.532151 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.532131    5318 scope.go:122] "RemoveContainer" containerID="eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26"
Nov 03 20:30:50.539096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.539079    5318 scope.go:122] "RemoveContainer" containerID="d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3"
Nov 03 20:30:50.545323 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.545303    5318 scope.go:122] "RemoveContainer" containerID="f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb"
Nov 03 20:30:50.551550 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.551535    5318 scope.go:122] "RemoveContainer" containerID="c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238"
Nov 03 20:30:50.557876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.557861    5318 scope.go:122] "RemoveContainer" containerID="6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed"
Nov 03 20:30:50.564177 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.564161    5318 scope.go:122] "RemoveContainer" containerID="9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e"
Nov 03 20:30:50.564515 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.564487    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e\": container with ID starting with 9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e not found: ID does not exist" containerID="9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e"
Nov 03 20:30:50.564576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.564528    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e"} err="failed to get container status \"9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e\": rpc error: code = NotFound desc = could not find container \"9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e\": container with ID starting with 9f61351a65cba832177c573fbc5a3947f2d727e1878b6457d87d693235fa257e not found: ID does not exist"
Nov 03 20:30:50.564576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.564553    5318 scope.go:122] "RemoveContainer" containerID="2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4"
Nov 03 20:30:50.564850 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.564832    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4\": container with ID starting with 2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4 not found: ID does not exist" containerID="2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4"
Nov 03 20:30:50.564910 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.564856    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4"} err="failed to get container status \"2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4\": rpc error: code = NotFound desc = could not find container \"2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4\": container with ID starting with 2f79a96473baa7526d2cce2d8410d758363f5cfd6db58a3b1d5956cf365f1ee4 not found: ID does not exist"
Nov 03 20:30:50.564910 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.564872    5318 scope.go:122] "RemoveContainer" containerID="eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26"
Nov 03 20:30:50.565115 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.565100    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26\": container with ID starting with eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26 not found: ID does not exist" containerID="eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26"
Nov 03 20:30:50.565163 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565119    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26"} err="failed to get container status \"eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26\": rpc error: code = NotFound desc = could not find container \"eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26\": container with ID starting with eea14ead054483b61e29f8bce0190a5a032dc3cb11e1cc6a914846545ad1ae26 not found: ID does not exist"
Nov 03 20:30:50.565163 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565131    5318 scope.go:122] "RemoveContainer" containerID="d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3"
Nov 03 20:30:50.565372 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.565331    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3\": container with ID starting with d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3 not found: ID does not exist" containerID="d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3"
Nov 03 20:30:50.565433 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565379    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3"} err="failed to get container status \"d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3\": rpc error: code = NotFound desc = could not find container \"d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3\": container with ID starting with d6895d585bc05449259e4f797cfaa541bf21b2bc5b68f55144a366002981ebb3 not found: ID does not exist"
Nov 03 20:30:50.565433 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565394    5318 scope.go:122] "RemoveContainer" containerID="f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb"
Nov 03 20:30:50.565642 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.565620    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb\": container with ID starting with f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb not found: ID does not exist" containerID="f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb"
Nov 03 20:30:50.565729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565651    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb"} err="failed to get container status \"f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb\": rpc error: code = NotFound desc = could not find container \"f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb\": container with ID starting with f2fcba4e1f35a5f94e1192085e9ea9e299c9cd1d0ef25daab6fc3fa3d1af76cb not found: ID does not exist"
Nov 03 20:30:50.565729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565673    5318 scope.go:122] "RemoveContainer" containerID="c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238"
Nov 03 20:30:50.565883 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.565868    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238\": container with ID starting with c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238 not found: ID does not exist" containerID="c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238"
Nov 03 20:30:50.565929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565886    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238"} err="failed to get container status \"c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238\": rpc error: code = NotFound desc = could not find container \"c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238\": container with ID starting with c914b84955c4bdce40f45fa99ccbbe80f5fed68a4d49aae112f70898b2829238 not found: ID does not exist"
Nov 03 20:30:50.565929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.565898    5318 scope.go:122] "RemoveContainer" containerID="6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed"
Nov 03 20:30:50.566063 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:50.566048    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed\": container with ID starting with 6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed not found: ID does not exist" containerID="6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed"
Nov 03 20:30:50.566098 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.566067    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed"} err="failed to get container status \"6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed\": rpc error: code = NotFound desc = could not find container \"6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed\": container with ID starting with 6d0bb11ca3f20c836bad68adc2a365cf693ab523edb6eb541036231fd47037ed not found: ID does not exist"
Nov 03 20:30:50.674014 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.673989    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:50.720315 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.720289    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:50.720937 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.720913    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:50.721186 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721169    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="init-config-reloader"
Nov 03 20:30:50.721186 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721182    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="init-config-reloader"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721204    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-metric"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721210    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-metric"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721219    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-web"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721223    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-web"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721231    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="config-reloader"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721235    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="config-reloader"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721245    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721249    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721259    5318 cpu_manager.go:414] "containerMap: removing container" podUID="336018cd-e7cc-4f23-809a-8b67aebd020d" containerName="console"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721263    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336018cd-e7cc-4f23-809a-8b67aebd020d" containerName="console"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721269    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="prom-label-proxy"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721275    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="prom-label-proxy"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721281    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="alertmanager"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721286    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="alertmanager"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721328    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-web"
Nov 03 20:30:50.721321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721349    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="config-reloader"
Nov 03 20:30:50.721807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721356    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy"
Nov 03 20:30:50.721807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721362    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="336018cd-e7cc-4f23-809a-8b67aebd020d" containerName="console"
Nov 03 20:30:50.721807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721368    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="prom-label-proxy"
Nov 03 20:30:50.721807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721373    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="alertmanager"
Nov 03 20:30:50.721807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.721382    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" containerName="kube-rbac-proxy-metric"
Nov 03 20:30:50.725910 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.725893    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.775040 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.774960    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:30:50.775156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775066    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:30:50.775156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775082    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:30:50.775156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775065    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:30:50.775156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775135    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:30:50.775156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775138    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:30:50.775407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775178    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:30:50.775407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775204    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:30:50.775407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.775307    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hl5hz\""
Nov 03 20:30:50.813459 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813436    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813465    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813484    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813509    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f88fddc-5a96-486b-b774-85753bc6c023-config-out\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813667 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813651    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813678    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-web-config\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813695    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f88fddc-5a96-486b-b774-85753bc6c023-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813775 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813712    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f88fddc-5a96-486b-b774-85753bc6c023-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813775 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813754    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7jk\" (UniqueName: \"kubernetes.io/projected/1f88fddc-5a96-486b-b774-85753bc6c023-kube-api-access-vl7jk\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813789    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f88fddc-5a96-486b-b774-85753bc6c023-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813852    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813884    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.813951 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.813900    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f88fddc-5a96-486b-b774-85753bc6c023-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.821072 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.821054    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:30:50.914830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.914800    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.914830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.914830    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-web-config\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.914850    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f88fddc-5a96-486b-b774-85753bc6c023-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.914975    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f88fddc-5a96-486b-b774-85753bc6c023-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915004    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7jk\" (UniqueName: \"kubernetes.io/projected/1f88fddc-5a96-486b-b774-85753bc6c023-kube-api-access-vl7jk\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915021    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f88fddc-5a96-486b-b774-85753bc6c023-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915065    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915110    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915133    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f88fddc-5a96-486b-b774-85753bc6c023-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915162    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915192    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915216    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915238    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f88fddc-5a96-486b-b774-85753bc6c023-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.915405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.915259    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f88fddc-5a96-486b-b774-85753bc6c023-config-out\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.916049 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.916033    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f88fddc-5a96-486b-b774-85753bc6c023-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.916464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.916420    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f88fddc-5a96-486b-b774-85753bc6c023-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.917972    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f88fddc-5a96-486b-b774-85753bc6c023-config-out\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.918439    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.918441    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918695 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.918555    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918758 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.918719    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918814 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.918786    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-web-config\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.918814 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.918804    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f88fddc-5a96-486b-b774-85753bc6c023-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.919376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.919359    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.919640 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.919625    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f88fddc-5a96-486b-b774-85753bc6c023-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:50.945387 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.945368    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:50.972297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:50.972277    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7jk\" (UniqueName: \"kubernetes.io/projected/1f88fddc-5a96-486b-b774-85753bc6c023-kube-api-access-vl7jk\") pod \"alertmanager-main-0\" (UID: \"1f88fddc-5a96-486b-b774-85753bc6c023\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:51.044131 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:51.044066    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:30:51.252426 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:51.252395    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9459b28f-1cb6-4b5f-b0df-f30ccd9f5470" path="/var/lib/kubelet/pods/9459b28f-1cb6-4b5f-b0df-f30ccd9f5470/volumes"
Nov 03 20:30:51.461306 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:51.461276    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:30:51.464146 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:51.464117    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f88fddc_5a96_486b_b774_85753bc6c023.slice/crio-4ae93fbd84de80eb74d5a6718d3a730494e8a71647509c20d9c0920488247793 WatchSource:0}: Error finding container 4ae93fbd84de80eb74d5a6718d3a730494e8a71647509c20d9c0920488247793: Status 404 returned error can't find the container with id 4ae93fbd84de80eb74d5a6718d3a730494e8a71647509c20d9c0920488247793
Nov 03 20:30:51.520164 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:51.520130    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"9d81c16a6b234cf72e8fe67eaab48fa9841b67ac3aa75e22227fa0dfedec5e46"}
Nov 03 20:30:51.520303 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:51.520207    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"4ae93fbd84de80eb74d5a6718d3a730494e8a71647509c20d9c0920488247793"}
Nov 03 20:30:52.526428 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:52.526392    5318 generic.go:356] "Generic (PLEG): container finished" podID="1f88fddc-5a96-486b-b774-85753bc6c023" containerID="9d81c16a6b234cf72e8fe67eaab48fa9841b67ac3aa75e22227fa0dfedec5e46" exitCode=0
Nov 03 20:30:52.526428 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:52.526430    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerDied","Data":"9d81c16a6b234cf72e8fe67eaab48fa9841b67ac3aa75e22227fa0dfedec5e46"}
Nov 03 20:30:53.535086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.535051    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"241467e869b2b467fd24410501da127d2e28281102c10cfa207cadada138f8e4"}
Nov 03 20:30:53.535471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.535120    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"8b0dbe8d6ed187c219ea0969ce94040a34bcb634e061307c8b0d188d4363c826"}
Nov 03 20:30:53.535471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.535152    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"5be5d4429f5fe88dac00e19518ca1c79580769323a24309219147dbe334b0b73"}
Nov 03 20:30:53.535471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.535181    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"a1d15440d4b458ac9cf29aa684f70e6b3df81d057278c0c352da169c719cd9e9"}
Nov 03 20:30:53.535471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.535206    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"b7e5ee9bf5f3fbf67bb4c10769370d68bb49e034bcb5c56cee1bb80921687167"}
Nov 03 20:30:53.535471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.535230    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f88fddc-5a96-486b-b774-85753bc6c023","Type":"ContainerStarted","Data":"052bd78dc7894b23e6c0a779b79e63d90ac84091a331e4d062ce95a5098a9f05"}
Nov 03 20:30:53.657004 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:53.656953    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.647138657 podStartE2EDuration="3.656938873s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.800216ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:53.656010517 +0000 UTC m=+46.635567357" watchObservedRunningTime="2026-11-03 20:30:53.656938873 +0000 UTC m=+46.636495714"
Nov 03 20:30:54.240959 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.240918    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:54.241538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.241221    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="prometheus" containerID="cri-o://fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42" gracePeriod=600
Nov 03 20:30:54.241538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.241251    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy" containerID="cri-o://4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f" gracePeriod=600
Nov 03 20:30:54.241538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.241277    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="thanos-sidecar" containerID="cri-o://6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1" gracePeriod=600
Nov 03 20:30:54.241538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.241402    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="config-reloader" containerID="cri-o://f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026" gracePeriod=600
Nov 03 20:30:54.241538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.241406    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-web" containerID="cri-o://89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3" gracePeriod=600
Nov 03 20:30:54.241538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.241414    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-thanos" containerID="cri-o://10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248" gracePeriod=600
Nov 03 20:30:54.390818 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.390774    5318 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 20:30:54.390944 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.390841    5318 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 20:30:54.390944 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.390867    5318 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 20:30:54.390944 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.390891    5318 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:30:54.488251 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.488225    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:54.541648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541586    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248" exitCode=0
Nov 03 20:30:54.541648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541614    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f" exitCode=0
Nov 03 20:30:54.541648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541623    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3" exitCode=0
Nov 03 20:30:54.541648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541632    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1" exitCode=0
Nov 03 20:30:54.541648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541642    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026" exitCode=0
Nov 03 20:30:54.541648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541650    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42" exitCode=0
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541654    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"}
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541673    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541749    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"}
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541780    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"}
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541811    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"}
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541830    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541836    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"}
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541943    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"}
Nov 03 20:30:54.542158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.541972    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6b7beedd-4ffd-432f-9a23-206aac61c107","Type":"ContainerDied","Data":"2f093cc341779b078e8a5979e9282247ac90a263a94e0d492019f27c46edd70b"}
Nov 03 20:30:54.549071 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.549056    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.555673 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555654    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.555770 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555709    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.555770 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555746    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.555935 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555776    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.555935 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555812    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.555935 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555830    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.555935 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555840    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.555935 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.555876    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.556292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556122    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556389    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556458    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556488    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556527    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556545    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556562    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556614    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556643    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556675    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556704    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556738    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556787    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556797    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556824    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.556859    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file\") pod \"6b7beedd-4ffd-432f-9a23-206aac61c107\" (UID: \"6b7beedd-4ffd-432f-9a23-206aac61c107\") "
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.557134    5318 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.557154    5318 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.557169    5318 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.557464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.557463    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.558475 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.557832    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.558475 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.557968    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:54.558475 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.558387    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.559067 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.558769    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:54.559444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.559409    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.559444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.559429    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:54.559746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.559714    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "kube-api-access-xlz5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:54.560268 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.560240    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:54.560598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.560570    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.560865 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.560834    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.561032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.561004    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.561032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.561020    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.561327 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.561307    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.561666 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.561644    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.561849 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.561828    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.566739 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.566720    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.570184 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.570162    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config" pod "6b7beedd-4ffd-432f-9a23-206aac61c107" (UID: "6b7beedd-4ffd-432f-9a23-206aac61c107"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:54.573149 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.573123    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.579482 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.579467    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.586212 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.586196    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.592329 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.592306    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.592593 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.592577    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.592637 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.592601    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"} err="failed to get container status \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist"
Nov 03 20:30:54.592637 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.592617    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.592852 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.592834    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.592891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.592862    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"} err="failed to get container status \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist"
Nov 03 20:30:54.592891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.592878    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.593146 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.593124    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.593188 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593152    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"} err="failed to get container status \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist"
Nov 03 20:30:54.593188 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593166    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.593422 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.593400    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.593469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593430    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"} err="failed to get container status \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist"
Nov 03 20:30:54.593469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593449    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.593690 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.593674    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.593732 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593697    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"} err="failed to get container status \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist"
Nov 03 20:30:54.593732 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593717    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.593926 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.593912    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.593968 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593929    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"} err="failed to get container status \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist"
Nov 03 20:30:54.593968 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.593941    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.594133 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:54.594117    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.594183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594136    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"} err="failed to get container status \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist"
Nov 03 20:30:54.594183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594147    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.594366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594351    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"} err="failed to get container status \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist"
Nov 03 20:30:54.594366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594366    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.594560 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594544    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"} err="failed to get container status \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist"
Nov 03 20:30:54.594599 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594560    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.594806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594784    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"} err="failed to get container status \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist"
Nov 03 20:30:54.594856 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594806    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.595014 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.594999    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"} err="failed to get container status \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist"
Nov 03 20:30:54.595014 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595013    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.595231 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595214    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"} err="failed to get container status \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist"
Nov 03 20:30:54.595279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595231    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.595493 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595470    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"} err="failed to get container status \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist"
Nov 03 20:30:54.595572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595494    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.595754 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595734    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"} err="failed to get container status \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist"
Nov 03 20:30:54.595808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595755    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.595984 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595966    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"} err="failed to get container status \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist"
Nov 03 20:30:54.596028 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.595992    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.596210 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596194    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"} err="failed to get container status \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist"
Nov 03 20:30:54.596255 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596209    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.596458 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596442    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"} err="failed to get container status \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist"
Nov 03 20:30:54.596458 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596457    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.596669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596654    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"} err="failed to get container status \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist"
Nov 03 20:30:54.596669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596668    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.596861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596846    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"} err="failed to get container status \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist"
Nov 03 20:30:54.596899 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.596861    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.597036 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597021    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"} err="failed to get container status \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist"
Nov 03 20:30:54.597073 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597035    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.597215 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597200    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"} err="failed to get container status \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist"
Nov 03 20:30:54.597255 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597215    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.597446 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597431    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"} err="failed to get container status \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist"
Nov 03 20:30:54.597494 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597446    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.597661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597645    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"} err="failed to get container status \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist"
Nov 03 20:30:54.597717 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597662    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.597862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597847    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"} err="failed to get container status \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist"
Nov 03 20:30:54.597910 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.597861    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.598044 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598028    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"} err="failed to get container status \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist"
Nov 03 20:30:54.598085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598044    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.598237 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598222    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"} err="failed to get container status \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist"
Nov 03 20:30:54.598292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598237    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.598475 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598457    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"} err="failed to get container status \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist"
Nov 03 20:30:54.598523 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598481    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.598684 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598668    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"} err="failed to get container status \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist"
Nov 03 20:30:54.598684 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598682    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.598883 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598867    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"} err="failed to get container status \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist"
Nov 03 20:30:54.598883 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.598882    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.599055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599040    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"} err="failed to get container status \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist"
Nov 03 20:30:54.599055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599054    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.599270 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599252    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"} err="failed to get container status \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist"
Nov 03 20:30:54.599322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599272    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.599521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599503    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"} err="failed to get container status \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist"
Nov 03 20:30:54.599563 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599521    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.599755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599734    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"} err="failed to get container status \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist"
Nov 03 20:30:54.599796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599756    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.599955 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599940    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"} err="failed to get container status \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist"
Nov 03 20:30:54.599993 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.599955    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.600149 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600132    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"} err="failed to get container status \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist"
Nov 03 20:30:54.600190 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600150    5318 scope.go:122] "RemoveContainer" containerID="10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"
Nov 03 20:30:54.600375 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600359    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248"} err="failed to get container status \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": rpc error: code = NotFound desc = could not find container \"10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248\": container with ID starting with 10597545ef67cf80dce080e60dc34ca9867a0002493b91786828ee77214f8248 not found: ID does not exist"
Nov 03 20:30:54.600422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600375    5318 scope.go:122] "RemoveContainer" containerID="4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"
Nov 03 20:30:54.600580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600558    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f"} err="failed to get container status \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": rpc error: code = NotFound desc = could not find container \"4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f\": container with ID starting with 4bf042ae9c2ea876107c43a94bfd8ce36b90526a1b9708141b98644efcdd227f not found: ID does not exist"
Nov 03 20:30:54.600618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600581    5318 scope.go:122] "RemoveContainer" containerID="89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"
Nov 03 20:30:54.600801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600783    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3"} err="failed to get container status \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": rpc error: code = NotFound desc = could not find container \"89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3\": container with ID starting with 89d15a1e484017faccf61895774f7a05d9dc47ce238782d781068c658f06d2f3 not found: ID does not exist"
Nov 03 20:30:54.600801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.600800    5318 scope.go:122] "RemoveContainer" containerID="6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"
Nov 03 20:30:54.601034 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601016    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1"} err="failed to get container status \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": rpc error: code = NotFound desc = could not find container \"6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1\": container with ID starting with 6ee8f7dfa6d09a8109a5a47e599ff6a82bb8619edccf64f9a738d3e2e6e99cd1 not found: ID does not exist"
Nov 03 20:30:54.601090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601034    5318 scope.go:122] "RemoveContainer" containerID="f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"
Nov 03 20:30:54.601253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601230    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026"} err="failed to get container status \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": rpc error: code = NotFound desc = could not find container \"f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026\": container with ID starting with f50a326b7fe001c06515a3ae2d03e5a15ed67b3de8af578828a8230e28d14026 not found: ID does not exist"
Nov 03 20:30:54.601253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601248    5318 scope.go:122] "RemoveContainer" containerID="fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"
Nov 03 20:30:54.601486 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601471    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42"} err="failed to get container status \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": rpc error: code = NotFound desc = could not find container \"fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42\": container with ID starting with fe9c1e9d26c0b64a5d025ba471a9e0a86c94b94aecbaa000bceef4bb52e23c42 not found: ID does not exist"
Nov 03 20:30:54.601486 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601485    5318 scope.go:122] "RemoveContainer" containerID="21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"
Nov 03 20:30:54.601679 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.601662    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a"} err="failed to get container status \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": rpc error: code = NotFound desc = could not find container \"21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a\": container with ID starting with 21e78fbbb16a1a924ace508755d8d22f1c5868bac7c481ac43c15cffaf7e021a not found: ID does not exist"
Nov 03 20:30:54.658706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658680    5318 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-kube-rbac-proxy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658707    5318 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658724    5318 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-metrics-client-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658739    5318 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658752    5318 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658765    5318 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-metrics-client-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658777    5318 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-grpc-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658790    5318 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658802    5318 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658815    5318 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-config-out\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658827    5318 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-tls-assets\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658840    5318 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-db\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658855    5318 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658868    5318 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-web-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658882    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlz5z\" (UniqueName: \"kubernetes.io/projected/6b7beedd-4ffd-432f-9a23-206aac61c107-kube-api-access-xlz5z\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658898    5318 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b7beedd-4ffd-432f-9a23-206aac61c107-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.658961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.658913    5318 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7beedd-4ffd-432f-9a23-206aac61c107-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:54.956988 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.956959    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:54.961052 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.961024    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:54.992170 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992146    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:54.992449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992436    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="config-reloader"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992450    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="config-reloader"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992465    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-thanos"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992470    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-thanos"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992482    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992487    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992497    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="prometheus"
Nov 03 20:30:54.992501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992502    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="prometheus"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992510    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="init-config-reloader"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992514    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="init-config-reloader"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992520    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="thanos-sidecar"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992524    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="thanos-sidecar"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992539    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-web"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992544    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-web"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992591    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-web"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992599    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="config-reloader"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992606    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy-thanos"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992612    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="kube-rbac-proxy"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992618    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="prometheus"
Nov 03 20:30:54.992704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.992623    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" containerName="thanos-sidecar"
Nov 03 20:30:54.996838 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:54.996823    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.046417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046390    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:30:55.046583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046567    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:30:55.046673 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046652    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:30:55.046673 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046670    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:30:55.046788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046657    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:30:55.046788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046688    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:30:55.046788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046693    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:30:55.046788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046704    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:30:55.046788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046712    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:30:55.046977 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046812    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:30:55.046977 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.046866    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:30:55.047263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.047247    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1gjsachoofh47\""
Nov 03 20:30:55.047406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.047393    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:30:55.047463 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.047422    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w7ljs\""
Nov 03 20:30:55.062289 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062260    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062383 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062299    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062383 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062320    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062399    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062501    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062555 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062520    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062555 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062539    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-config\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062632 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062561    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzrn\" (UniqueName: \"kubernetes.io/projected/b4df91e9-57b1-42d3-9308-dd68df3223c8-kube-api-access-qnzrn\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062632 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062604    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4df91e9-57b1-42d3-9308-dd68df3223c8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062719 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062643    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062719 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062681    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062798 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062724    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062832 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062803    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062832 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062825    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4df91e9-57b1-42d3-9308-dd68df3223c8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062903 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062840    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062903 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062854    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.062903 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062868    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.063000 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062926    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.063000 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062945    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.063000 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.062972    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.093447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.093425    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:30:55.093626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.093609    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:30:55.164107 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164044    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164107 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164081    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4df91e9-57b1-42d3-9308-dd68df3223c8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164107    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164131    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164233    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164273    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164300    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164356    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164390    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164431    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164461    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164503    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164533    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164558    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164590    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-config\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164637    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzrn\" (UniqueName: \"kubernetes.io/projected/b4df91e9-57b1-42d3-9308-dd68df3223c8-kube-api-access-qnzrn\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164663    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4df91e9-57b1-42d3-9308-dd68df3223c8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164684    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.164720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164710    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.165161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164733    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.165161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164781    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.165161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.164842    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.165646 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.165370    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.166050 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.166026    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.166139 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.166038    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.166139 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.166114    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.166534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.166495    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.168707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.168397    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4df91e9-57b1-42d3-9308-dd68df3223c8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.168707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.168501    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4df91e9-57b1-42d3-9308-dd68df3223c8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.168707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.168616    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.168928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.168724    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.168928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.168802    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.169030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.169004    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.169120 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.169101    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.169233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.169206    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4df91e9-57b1-42d3-9308-dd68df3223c8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.169440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.169419    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-config\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.169933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.169907    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.170002 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.169967    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.170126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.170111    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4df91e9-57b1-42d3-9308-dd68df3223c8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.208090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.208067    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:55.224446 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.224421    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzrn\" (UniqueName: \"kubernetes.io/projected/b4df91e9-57b1-42d3-9308-dd68df3223c8-kube-api-access-qnzrn\") pod \"prometheus-k8s-0\" (UID: \"b4df91e9-57b1-42d3-9308-dd68df3223c8\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.255056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.255029    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7beedd-4ffd-432f-9a23-206aac61c107" path="/var/lib/kubelet/pods/6b7beedd-4ffd-432f-9a23-206aac61c107/volumes"
Nov 03 20:30:55.318577 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.318552    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:30:55.738825 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:55.738798    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:30:55.741383 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:30:55.741351    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4df91e9_57b1_42d3_9308_dd68df3223c8.slice/crio-2166823536f82da7137291222302a2a1ffdd73d6eba44678ab8c3d7999e1a27c WatchSource:0}: Error finding container 2166823536f82da7137291222302a2a1ffdd73d6eba44678ab8c3d7999e1a27c: Status 404 returned error can't find the container with id 2166823536f82da7137291222302a2a1ffdd73d6eba44678ab8c3d7999e1a27c
Nov 03 20:30:55.931772 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:30:55.931740    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:30:56.555702 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:56.555672    5318 generic.go:356] "Generic (PLEG): container finished" podID="b4df91e9-57b1-42d3-9308-dd68df3223c8" containerID="5954876044a827543edf5f1a7c452c8c6de17d34d448ced903a04a5ab71ad30c" exitCode=0
Nov 03 20:30:56.555858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:56.555759    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerDied","Data":"5954876044a827543edf5f1a7c452c8c6de17d34d448ced903a04a5ab71ad30c"}
Nov 03 20:30:56.555858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:56.555828    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"2166823536f82da7137291222302a2a1ffdd73d6eba44678ab8c3d7999e1a27c"}
Nov 03 20:30:57.562753 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.562720    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"ea70dcb1840a17cabcbccc1f7562490ca382178e28b1fb56e964607d1485ad7b"}
Nov 03 20:30:57.563115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.562807    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"44e31adea2fb0a1095d6ce3b7afbf0a7222f2ad9e36ed775c7aad059958af0d4"}
Nov 03 20:30:57.563115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.562844    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"f5c6a73fe3765ed8f2dc36291b3c489d3780ab206eec2eb9adf04040f5e68249"}
Nov 03 20:30:57.563115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.562869    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"7dc44d4798c14082bba52329375e2136ce981644079a97061973c68ae74b5821"}
Nov 03 20:30:57.563115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.562891    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"665d32057a8d35b4910c218f0993ae2a780397e45b4bb92cbe16e97f874921a3"}
Nov 03 20:30:57.563115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.562914    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4df91e9-57b1-42d3-9308-dd68df3223c8","Type":"ContainerStarted","Data":"1cef9194e88f2a382c2914008f5308a27fc8a18b48dca79d776dcd8ea28bfd5d"}
Nov 03 20:30:57.688748 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:57.688697    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.673073124 podStartE2EDuration="3.688681585s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.608461ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:57.688140323 +0000 UTC m=+50.667697162" watchObservedRunningTime="2026-11-03 20:30:57.688681585 +0000 UTC m=+50.668238426"
Nov 03 20:30:59.475378 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:59.475327    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:59.667466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:59.667425    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:59.862703 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:30:59.862667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:00.319554 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:00.319483    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:31:09.763681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:09.763645    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c584b584d-v26mn"]
Nov 03 20:31:10.367246 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:10.367207    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:31:10.382619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:10.382591    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:31:10.627505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:10.627435    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:31:13.221485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:13.221445    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:14.162587 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:14.162545    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:14.367180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:14.367151    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:25.932096 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:31:25.932061    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:31:26.021946 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:26.021918    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:26.221296 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:26.221221    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:26.421733 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:26.421700    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:26.871802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:26.871768    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:27.068686 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:27.068644    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:32.470471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:32.470430    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:32.682261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:32.682203    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:34.782497 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:34.782439    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c584b584d-v26mn" podUID="917ad3c9-eea6-4f27-b366-26fd14c8ab0d" containerName="console" containerID="cri-o://71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf" gracePeriod=15
Nov 03 20:31:35.034916 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.034866    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:31:35.095133 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095108    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095143    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095174    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095215    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095371    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095428    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095549 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095476    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw\" (UniqueName: \"kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095549 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095510    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle\") pod \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\" (UID: \"917ad3c9-eea6-4f27-b366-26fd14c8ab0d\") "
Nov 03 20:31:35.095549 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095530    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:35.095703 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095670    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:35.095759 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095723    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:35.095835 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095815    5318 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-tmp\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.095957 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095840    5318 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-oauth-serving-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.095957 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095884    5318 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.095957 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.095945    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:35.096126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.096021    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:35.097349 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.097316    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:35.097535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.097517    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:35.097583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.097557    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw" pod "917ad3c9-eea6-4f27-b366-26fd14c8ab0d" (UID: "917ad3c9-eea6-4f27-b366-26fd14c8ab0d"). InnerVolumeSpecName "kube-api-access-dxdnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:35.197225 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.197203    5318 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-oauth-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.197225 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.197223    5318 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-config\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.197366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.197234    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdnw\" (UniqueName: \"kubernetes.io/projected/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-kube-api-access-dxdnw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.197366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.197245    5318 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-trusted-ca-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.197366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.197256    5318 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/917ad3c9-eea6-4f27-b366-26fd14c8ab0d-console-serving-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.704559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.704521    5318 generic.go:356] "Generic (PLEG): container finished" podID="917ad3c9-eea6-4f27-b366-26fd14c8ab0d" containerID="71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf" exitCode=0
Nov 03 20:31:35.704720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.704568    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584b584d-v26mn"
Nov 03 20:31:35.704720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.704602    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584b584d-v26mn" event={"ID":"917ad3c9-eea6-4f27-b366-26fd14c8ab0d","Type":"ContainerDied","Data":"71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf"}
Nov 03 20:31:35.704720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.704650    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584b584d-v26mn" event={"ID":"917ad3c9-eea6-4f27-b366-26fd14c8ab0d","Type":"ContainerDied","Data":"363e071b50d76c5306d5fbe7977129abcaa69703fa0fdb948c81bccc9aa774c3"}
Nov 03 20:31:35.704720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.704670    5318 scope.go:122] "RemoveContainer" containerID="71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf"
Nov 03 20:31:35.712140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.712127    5318 scope.go:122] "RemoveContainer" containerID="71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf"
Nov 03 20:31:35.712433 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:31:35.712409    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf\": container with ID starting with 71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf not found: ID does not exist" containerID="71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf"
Nov 03 20:31:35.712480 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.712433    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf"} err="failed to get container status \"71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf\": rpc error: code = NotFound desc = could not find container \"71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf\": container with ID starting with 71e46da5d626bdf90e7d2a41967d5fe0365ae92ec4b715a221042337328c3ccf not found: ID does not exist"
Nov 03 20:31:35.857277 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.857257    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c584b584d-v26mn"]
Nov 03 20:31:35.903256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:35.903233    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c584b584d-v26mn"]
Nov 03 20:31:37.252617 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:37.252583    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917ad3c9-eea6-4f27-b366-26fd14c8ab0d" path="/var/lib/kubelet/pods/917ad3c9-eea6-4f27-b366-26fd14c8ab0d/volumes"
Nov 03 20:31:55.107316 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:31:55.107278    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:55.931662 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:31:55.931628    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:32:13.546424 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:32:13.546383    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:25.931271 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:32:25.931235    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:32:55.934244 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:32:55.934186    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:33:13.872986 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:33:13.872952    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:25.932943 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:33:25.932859    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:33:55.932702 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:33:55.932663    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:34:00.063115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:00.063076    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:02.073589 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:02.073553    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:14.220067 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:14.220027    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:25.932378 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:34:25.932325    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:34:26.756906 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:26.756872    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:26.954395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:26.954364    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:35.185407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:35.185365    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:41.953825 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:41.953790    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:42.169769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:42.169737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:43.599360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:43.598796    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:43.795035 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:43.794999    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:44.914496 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:44.914458    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:51.683958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:51.683884    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:51.888649 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:51.888621    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.115758 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.115728    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.672915 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.672879    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.780128 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.780097    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"]
Nov 03 20:34:52.780487 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.780421    5318 cpu_manager.go:414] "containerMap: removing container" podUID="917ad3c9-eea6-4f27-b366-26fd14c8ab0d" containerName="console"
Nov 03 20:34:52.780487 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.780432    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917ad3c9-eea6-4f27-b366-26fd14c8ab0d" containerName="console"
Nov 03 20:34:52.780563 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.780501    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="917ad3c9-eea6-4f27-b366-26fd14c8ab0d" containerName="console"
Nov 03 20:34:52.783515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.783500    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.813629 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.813608    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"]
Nov 03 20:34:52.816825 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.816811    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:52.835192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.835175    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:34:52.858931 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.858910    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:34:52.859362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.859348    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kzbcz\""
Nov 03 20:34:52.860018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.860000    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"]
Nov 03 20:34:52.883158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.883131    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"]
Nov 03 20:34:52.887015 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.886993    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.887133 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.887040    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.887133 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.887080    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"]
Nov 03 20:34:52.887133 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.887117    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdw6q\" (UniqueName: \"kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.887302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.887187    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:52.890082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.890063    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pxsq5\""
Nov 03 20:34:52.909426 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.909406    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"]
Nov 03 20:34:52.938389 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.938327    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-2gmr6\""
Nov 03 20:34:52.987902 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.987874    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:52.987997 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.987947    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.987997 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.987974    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:52.988105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.987995    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdw6q\" (UniqueName: \"kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.988105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988023    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6nl4\" (UniqueName: \"kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:52.988182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988136    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:52.988282 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988263    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvsm\" (UniqueName: \"kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:52.988318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988298    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.988377 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988322    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:52.988539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988520    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.988721 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.988702    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:52.997288 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:52.997265    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.045253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.045233    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdw6q\" (UniqueName: \"kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:53.088985 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.088962    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.089064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.088991    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6nl4\" (UniqueName: \"kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.089064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089010    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.089064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089054    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvsm\" (UniqueName: \"kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.089176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089089    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.089176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089133    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.089406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089385    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.089451 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089413    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.089493 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089475    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.089528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.089505    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.099022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.099005    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:34:53.146757 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.146674    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6nl4\" (UniqueName: \"kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.147212 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.146820    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvsm\" (UniqueName: \"kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.197831 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.197755    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.202642 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.202620    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:34:53.393538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.393511    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.430656 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.430625    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:34:53.464239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.464170    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"]
Nov 03 20:34:53.468881 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:34:53.468849    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ef2ce6_002b_40ee_a476_1917fc087fa0.slice/crio-3240531ae8cc7cff98f236023a60816f08e5b6d5d620da9c52eecbc0eab3c702 WatchSource:0}: Error finding container 3240531ae8cc7cff98f236023a60816f08e5b6d5d620da9c52eecbc0eab3c702: Status 404 returned error can't find the container with id 3240531ae8cc7cff98f236023a60816f08e5b6d5d620da9c52eecbc0eab3c702
Nov 03 20:34:53.566712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.566680    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"]
Nov 03 20:34:53.569637 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:34:53.569610    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026cddfe_01c0_4475_9281_38a71994bf79.slice/crio-95110888586dab698e00a96db8c73d21b836dc911376c6f6079819f0b47eebab WatchSource:0}: Error finding container 95110888586dab698e00a96db8c73d21b836dc911376c6f6079819f0b47eebab: Status 404 returned error can't find the container with id 95110888586dab698e00a96db8c73d21b836dc911376c6f6079819f0b47eebab
Nov 03 20:34:53.797390 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:53.797297    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"]
Nov 03 20:34:53.800204 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:34:53.800177    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d86e3fa_96b1_4870_8df7_508ea15b4c9b.slice/crio-b2f271518e1798fd10f5a51959333b8b7f0f735ae18f4477688f53c4ffb12d64 WatchSource:0}: Error finding container b2f271518e1798fd10f5a51959333b8b7f0f735ae18f4477688f53c4ffb12d64: Status 404 returned error can't find the container with id b2f271518e1798fd10f5a51959333b8b7f0f735ae18f4477688f53c4ffb12d64
Nov 03 20:34:54.405962 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:54.405931    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm" event={"ID":"c4ef2ce6-002b-40ee-a476-1917fc087fa0","Type":"ContainerStarted","Data":"3240531ae8cc7cff98f236023a60816f08e5b6d5d620da9c52eecbc0eab3c702"}
Nov 03 20:34:54.406950 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:54.406927    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" event={"ID":"3d86e3fa-96b1-4870-8df7-508ea15b4c9b","Type":"ContainerStarted","Data":"b2f271518e1798fd10f5a51959333b8b7f0f735ae18f4477688f53c4ffb12d64"}
Nov 03 20:34:54.407847 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:54.407824    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585" event={"ID":"026cddfe-01c0-4475-9281-38a71994bf79","Type":"ContainerStarted","Data":"95110888586dab698e00a96db8c73d21b836dc911376c6f6079819f0b47eebab"}
Nov 03 20:34:55.942537 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:34:55.942500    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:34:57.102886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:34:57.102846    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:01.444025 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:01.443552    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585" event={"ID":"026cddfe-01c0-4475-9281-38a71994bf79","Type":"ContainerStarted","Data":"9a64f168848ac4490eb95f89b7dd61d4741cd805a1c5d1b8524dfb176587759b"}
Nov 03 20:35:01.445514 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:01.445480    5318 generic.go:356] "Generic (PLEG): container finished" podID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerID="8dc7b0d2e06a0c908ce67e6c9058675b2f12f765451b5080a811bf83f16e01c8" exitCode=0
Nov 03 20:35:01.445625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:01.445533    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm" event={"ID":"c4ef2ce6-002b-40ee-a476-1917fc087fa0","Type":"ContainerDied","Data":"8dc7b0d2e06a0c908ce67e6c9058675b2f12f765451b5080a811bf83f16e01c8"}
Nov 03 20:35:02.451089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:02.451053    5318 generic.go:356] "Generic (PLEG): container finished" podID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerID="feb39811609c227eba1dbf991ceb34bb8c89af1da04cfcb3c148044ce6e0949e" exitCode=0
Nov 03 20:35:02.451575 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:02.451138    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" event={"ID":"3d86e3fa-96b1-4870-8df7-508ea15b4c9b","Type":"ContainerDied","Data":"feb39811609c227eba1dbf991ceb34bb8c89af1da04cfcb3c148044ce6e0949e"}
Nov 03 20:35:02.452698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:02.452678    5318 generic.go:356] "Generic (PLEG): container finished" podID="026cddfe-01c0-4475-9281-38a71994bf79" containerID="9a64f168848ac4490eb95f89b7dd61d4741cd805a1c5d1b8524dfb176587759b" exitCode=0
Nov 03 20:35:02.452793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:02.452743    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585" event={"ID":"026cddfe-01c0-4475-9281-38a71994bf79","Type":"ContainerDied","Data":"9a64f168848ac4490eb95f89b7dd61d4741cd805a1c5d1b8524dfb176587759b"}
Nov 03 20:35:03.458367 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:03.458323    5318 generic.go:356] "Generic (PLEG): container finished" podID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerID="ed0d0fa61563add7e87ecedde9e0277dd54a08c692a890cc0d3d0069da08efa5" exitCode=0
Nov 03 20:35:03.458763 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:03.458381    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm" event={"ID":"c4ef2ce6-002b-40ee-a476-1917fc087fa0","Type":"ContainerDied","Data":"ed0d0fa61563add7e87ecedde9e0277dd54a08c692a890cc0d3d0069da08efa5"}
Nov 03 20:35:04.463797 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:04.463714    5318 generic.go:356] "Generic (PLEG): container finished" podID="026cddfe-01c0-4475-9281-38a71994bf79" containerID="34a906ce7b34ef5376be65f3f19fec99d8353c49b7e9d1383aa302f449af9acf" exitCode=0
Nov 03 20:35:04.464219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:04.463808    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585" event={"ID":"026cddfe-01c0-4475-9281-38a71994bf79","Type":"ContainerDied","Data":"34a906ce7b34ef5376be65f3f19fec99d8353c49b7e9d1383aa302f449af9acf"}
Nov 03 20:35:04.465792 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:04.465762    5318 generic.go:356] "Generic (PLEG): container finished" podID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerID="f71dd877088b76e35881fc4a251c76da276fe9751893a467185835811e724842" exitCode=0
Nov 03 20:35:04.465867 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:04.465793    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" event={"ID":"3d86e3fa-96b1-4870-8df7-508ea15b4c9b","Type":"ContainerDied","Data":"f71dd877088b76e35881fc4a251c76da276fe9751893a467185835811e724842"}
Nov 03 20:35:07.333433 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:07.333393    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:35:07.335108 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:07.335083    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:35:12.514905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:12.514819    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" event={"ID":"3d86e3fa-96b1-4870-8df7-508ea15b4c9b","Type":"ContainerStarted","Data":"95fe4bb3c1eefe0cc2b5fa660dfcdd4a8a3ba7d2c74532dde82a8bbdddf1ebc3"}
Nov 03 20:35:12.627323 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:12.627262    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" podStartSLOduration=3.335877848 podStartE2EDuration="20.62724369s" totalImagesPullingTime="17.291365842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:12.62656788 +0000 UTC m=+305.606124721" watchObservedRunningTime="2026-11-03 20:35:12.62724369 +0000 UTC m=+305.606800532"
Nov 03 20:35:13.522821 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:13.522789    5318 generic.go:356] "Generic (PLEG): container finished" podID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerID="95fe4bb3c1eefe0cc2b5fa660dfcdd4a8a3ba7d2c74532dde82a8bbdddf1ebc3" exitCode=0
Nov 03 20:35:13.523251 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:13.522847    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" event={"ID":"3d86e3fa-96b1-4870-8df7-508ea15b4c9b","Type":"ContainerDied","Data":"95fe4bb3c1eefe0cc2b5fa660dfcdd4a8a3ba7d2c74532dde82a8bbdddf1ebc3"}
Nov 03 20:35:14.533245 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.533165    5318 generic.go:356] "Generic (PLEG): container finished" podID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerID="e476266a3f1df8dd1726a04225932c701147808c35bd7a87f1e60800dd307e11" exitCode=0
Nov 03 20:35:14.533683 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.533248    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm" event={"ID":"c4ef2ce6-002b-40ee-a476-1917fc087fa0","Type":"ContainerDied","Data":"e476266a3f1df8dd1726a04225932c701147808c35bd7a87f1e60800dd307e11"}
Nov 03 20:35:14.535299 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.535276    5318 generic.go:356] "Generic (PLEG): container finished" podID="026cddfe-01c0-4475-9281-38a71994bf79" containerID="361422f29dcb9210cb170c3a28daba125f700f6fffed677e15acb8cb681216d7" exitCode=0
Nov 03 20:35:14.535431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.535360    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585" event={"ID":"026cddfe-01c0-4475-9281-38a71994bf79","Type":"ContainerDied","Data":"361422f29dcb9210cb170c3a28daba125f700f6fffed677e15acb8cb681216d7"}
Nov 03 20:35:14.628978 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.628937    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:14.651783 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.651760    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:35:14.787113 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.787041    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4\" (UniqueName: \"kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4\") pod \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") "
Nov 03 20:35:14.787113 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.787084    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util\") pod \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") "
Nov 03 20:35:14.787326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.787135    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle\") pod \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\" (UID: \"3d86e3fa-96b1-4870-8df7-508ea15b4c9b\") "
Nov 03 20:35:14.787720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.787693    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle" pod "3d86e3fa-96b1-4870-8df7-508ea15b4c9b" (UID: "3d86e3fa-96b1-4870-8df7-508ea15b4c9b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:14.789302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.789282    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4" pod "3d86e3fa-96b1-4870-8df7-508ea15b4c9b" (UID: "3d86e3fa-96b1-4870-8df7-508ea15b4c9b"). InnerVolumeSpecName "kube-api-access-d6nl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:14.791421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.791398    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util" pod "3d86e3fa-96b1-4870-8df7-508ea15b4c9b" (UID: "3d86e3fa-96b1-4870-8df7-508ea15b4c9b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:14.887808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.887777    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6nl4\" (UniqueName: \"kubernetes.io/projected/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-kube-api-access-d6nl4\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:14.887808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.887805    5318 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-util\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:14.887808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:14.887814    5318 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d86e3fa-96b1-4870-8df7-508ea15b4c9b-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:15.541529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.541453    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq"
Nov 03 20:35:15.541529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.541473    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c876fq" event={"ID":"3d86e3fa-96b1-4870-8df7-508ea15b4c9b","Type":"ContainerDied","Data":"b2f271518e1798fd10f5a51959333b8b7f0f735ae18f4477688f53c4ffb12d64"}
Nov 03 20:35:15.541529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.541524    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f271518e1798fd10f5a51959333b8b7f0f735ae18f4477688f53c4ffb12d64"
Nov 03 20:35:15.686798 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.686773    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:35:15.692583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.692565    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:35:15.795970 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.795911    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util\") pod \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") "
Nov 03 20:35:15.795970 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.795957    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle\") pod \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") "
Nov 03 20:35:15.796144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.795978    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm\" (UniqueName: \"kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm\") pod \"026cddfe-01c0-4475-9281-38a71994bf79\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") "
Nov 03 20:35:15.796144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.796003    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle\") pod \"026cddfe-01c0-4475-9281-38a71994bf79\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") "
Nov 03 20:35:15.796144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.796046    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q\" (UniqueName: \"kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q\") pod \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\" (UID: \"c4ef2ce6-002b-40ee-a476-1917fc087fa0\") "
Nov 03 20:35:15.796144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.796093    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util\") pod \"026cddfe-01c0-4475-9281-38a71994bf79\" (UID: \"026cddfe-01c0-4475-9281-38a71994bf79\") "
Nov 03 20:35:15.797058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.796657    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle" pod "026cddfe-01c0-4475-9281-38a71994bf79" (UID: "026cddfe-01c0-4475-9281-38a71994bf79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:15.797058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.796860    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle" pod "c4ef2ce6-002b-40ee-a476-1917fc087fa0" (UID: "c4ef2ce6-002b-40ee-a476-1917fc087fa0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:15.798819 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.798783    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm" pod "026cddfe-01c0-4475-9281-38a71994bf79" (UID: "026cddfe-01c0-4475-9281-38a71994bf79"). InnerVolumeSpecName "kube-api-access-9zvsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:15.798908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.798836    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q" pod "c4ef2ce6-002b-40ee-a476-1917fc087fa0" (UID: "c4ef2ce6-002b-40ee-a476-1917fc087fa0"). InnerVolumeSpecName "kube-api-access-hdw6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:15.801390 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.801332    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util" pod "026cddfe-01c0-4475-9281-38a71994bf79" (UID: "026cddfe-01c0-4475-9281-38a71994bf79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:15.802875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.802853    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util" pod "c4ef2ce6-002b-40ee-a476-1917fc087fa0" (UID: "c4ef2ce6-002b-40ee-a476-1917fc087fa0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:15.896874 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.896849    5318 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-util\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:15.896874 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.896872    5318 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-util\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:15.897018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.896885    5318 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4ef2ce6-002b-40ee-a476-1917fc087fa0-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:15.897018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.896899    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvsm\" (UniqueName: \"kubernetes.io/projected/026cddfe-01c0-4475-9281-38a71994bf79-kube-api-access-9zvsm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:15.897018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.896912    5318 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/026cddfe-01c0-4475-9281-38a71994bf79-bundle\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:15.897018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:15.896926    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdw6q\" (UniqueName: \"kubernetes.io/projected/c4ef2ce6-002b-40ee-a476-1917fc087fa0-kube-api-access-hdw6q\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:16.547425 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:16.547394    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm"
Nov 03 20:35:16.547425 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:16.547415    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax62nm" event={"ID":"c4ef2ce6-002b-40ee-a476-1917fc087fa0","Type":"ContainerDied","Data":"3240531ae8cc7cff98f236023a60816f08e5b6d5d620da9c52eecbc0eab3c702"}
Nov 03 20:35:16.547962 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:16.547468    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3240531ae8cc7cff98f236023a60816f08e5b6d5d620da9c52eecbc0eab3c702"
Nov 03 20:35:16.549318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:16.549294    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585" event={"ID":"026cddfe-01c0-4475-9281-38a71994bf79","Type":"ContainerDied","Data":"95110888586dab698e00a96db8c73d21b836dc911376c6f6079819f0b47eebab"}
Nov 03 20:35:16.549318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:16.549317    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hp585"
Nov 03 20:35:16.549528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:16.549326    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95110888586dab698e00a96db8c73d21b836dc911376c6f6079819f0b47eebab"
Nov 03 20:35:19.612314 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:19.612275    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:19.816144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:19.816118    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:25.932121 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:35:25.932077    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:35:28.567916 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:28.567885    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:32.157972 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:32.157929    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:34.206058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:34.206026    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:34.501992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:34.501909    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.366804 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.366774    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.575723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.575693    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.674407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674325    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"]
Nov 03 20:35:35.674648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674636    5318 cpu_manager.go:414] "containerMap: removing container" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="util"
Nov 03 20:35:35.674692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674650    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="util"
Nov 03 20:35:35.674692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674664    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="pull"
Nov 03 20:35:35.674692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674669    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="pull"
Nov 03 20:35:35.674692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674681    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="util"
Nov 03 20:35:35.674692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674686    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="util"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674694    5318 cpu_manager.go:414] "containerMap: removing container" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674699    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674705    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674709    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674722    5318 cpu_manager.go:414] "containerMap: removing container" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="util"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674726    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="util"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674733    5318 cpu_manager.go:414] "containerMap: removing container" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="pull"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674737    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="pull"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674756    5318 cpu_manager.go:414] "containerMap: removing container" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="pull"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674760    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="pull"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674767    5318 cpu_manager.go:414] "containerMap: removing container" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674771    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674811    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ef2ce6-002b-40ee-a476-1917fc087fa0" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674823    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d86e3fa-96b1-4870-8df7-508ea15b4c9b" containerName="extract"
Nov 03 20:35:35.674841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.674830    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="026cddfe-01c0-4475-9281-38a71994bf79" containerName="extract"
Nov 03 20:35:35.678156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.678139    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.727584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.727562    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:35:35.730538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.730519    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:35:35.750318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.750299    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.750432 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.750329    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.750432 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.750363    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6185bc78-db05-49ac-a333-426e95fcba93-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.750525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.750450    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbspq\" (UniqueName: \"kubernetes.io/projected/6185bc78-db05-49ac-a333-426e95fcba93-kube-api-access-fbspq\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.750525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.750505    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6185bc78-db05-49ac-a333-426e95fcba93-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.750598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.750540    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.774855 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.774829    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"]
Nov 03 20:35:35.820176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.820159    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:35:35.820378 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.820355    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:35:35.820448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.820405    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:35:35.830220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.830200    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fplxc\""
Nov 03 20:35:35.851199 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851178    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.851297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851208    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.851297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851225    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6185bc78-db05-49ac-a333-426e95fcba93-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.851297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851246    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbspq\" (UniqueName: \"kubernetes.io/projected/6185bc78-db05-49ac-a333-426e95fcba93-kube-api-access-fbspq\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.851297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851290    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6185bc78-db05-49ac-a333-426e95fcba93-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.851544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851313    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.851681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.851658    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6185bc78-db05-49ac-a333-426e95fcba93-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.853722 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.853699    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6185bc78-db05-49ac-a333-426e95fcba93-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.853843 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.853778    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.853843 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.853821    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.853974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.853953    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6185bc78-db05-49ac-a333-426e95fcba93-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.913418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.913400    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbspq\" (UniqueName: \"kubernetes.io/projected/6185bc78-db05-49ac-a333-426e95fcba93-kube-api-access-fbspq\") pod \"opendatahub-operator-controller-manager-6f448d84c4-zz68f\" (UID: \"6185bc78-db05-49ac-a333-426e95fcba93\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:35.976085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.976024    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.992190 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:35.992166    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:36.199069 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.199035    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.407274 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.407242    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.504918 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.504896    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"]
Nov 03 20:35:36.508157 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:35:36.508133    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6185bc78_db05_49ac_a333_426e95fcba93.slice/crio-154b6810f3ccd03db6e86af4cb7ae1f65a57205145489b17977c88229330635c WatchSource:0}: Error finding container 154b6810f3ccd03db6e86af4cb7ae1f65a57205145489b17977c88229330635c: Status 404 returned error can't find the container with id 154b6810f3ccd03db6e86af4cb7ae1f65a57205145489b17977c88229330635c
Nov 03 20:35:36.509817 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.509801    5318 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:35:36.625510 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.625484    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f" event={"ID":"6185bc78-db05-49ac-a333-426e95fcba93","Type":"ContainerStarted","Data":"154b6810f3ccd03db6e86af4cb7ae1f65a57205145489b17977c88229330635c"}
Nov 03 20:35:36.791594 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.791536    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.990837 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:36.990798    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.718844 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.718798    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"]
Nov 03 20:35:39.722201 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.722180    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:39.774099 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.774078    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-d6gkk\""
Nov 03 20:35:39.822217 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.822193    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"]
Nov 03 20:35:39.848550 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.848521    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 20:35:39.848658 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.848587    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:35:39.884663 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.884636    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfnx\" (UniqueName: \"kubernetes.io/projected/db8f8416-28df-478a-9197-b03d494ebaf9-kube-api-access-khfnx\") pod \"jobset-operator-674cd59dc5-cflnq\" (UID: \"db8f8416-28df-478a-9197-b03d494ebaf9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:39.884765 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.884687    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8f8416-28df-478a-9197-b03d494ebaf9-tmp\") pod \"jobset-operator-674cd59dc5-cflnq\" (UID: \"db8f8416-28df-478a-9197-b03d494ebaf9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:39.986030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.985967    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfnx\" (UniqueName: \"kubernetes.io/projected/db8f8416-28df-478a-9197-b03d494ebaf9-kube-api-access-khfnx\") pod \"jobset-operator-674cd59dc5-cflnq\" (UID: \"db8f8416-28df-478a-9197-b03d494ebaf9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:39.986030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.986014    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8f8416-28df-478a-9197-b03d494ebaf9-tmp\") pod \"jobset-operator-674cd59dc5-cflnq\" (UID: \"db8f8416-28df-478a-9197-b03d494ebaf9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:39.986397 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:39.986379    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db8f8416-28df-478a-9197-b03d494ebaf9-tmp\") pod \"jobset-operator-674cd59dc5-cflnq\" (UID: \"db8f8416-28df-478a-9197-b03d494ebaf9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:40.047128 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.047105    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfnx\" (UniqueName: \"kubernetes.io/projected/db8f8416-28df-478a-9197-b03d494ebaf9-kube-api-access-khfnx\") pod \"jobset-operator-674cd59dc5-cflnq\" (UID: \"db8f8416-28df-478a-9197-b03d494ebaf9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:40.344872 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.344846    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"
Nov 03 20:35:40.509225 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.509194    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:40.645529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.645462    5318 generic.go:356] "Generic (PLEG): container finished" podID="6185bc78-db05-49ac-a333-426e95fcba93" containerID="fad161af9816afd52bf124fad24383b0e01367c9ee531543a8486c6754e5dfca" exitCode=0
Nov 03 20:35:40.645529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.645523    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f" event={"ID":"6185bc78-db05-49ac-a333-426e95fcba93","Type":"ContainerDied","Data":"fad161af9816afd52bf124fad24383b0e01367c9ee531543a8486c6754e5dfca"}
Nov 03 20:35:40.706158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.706134    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:40.708518 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:35:40.708487    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb8f8416_28df_478a_9197_b03d494ebaf9.slice/crio-520dd48fbec2f4875e111f75a42fa29686c7dc4618648644cb1a0c54a76149ef WatchSource:0}: Error finding container 520dd48fbec2f4875e111f75a42fa29686c7dc4618648644cb1a0c54a76149ef: Status 404 returned error can't find the container with id 520dd48fbec2f4875e111f75a42fa29686c7dc4618648644cb1a0c54a76149ef
Nov 03 20:35:40.741394 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:40.741374    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq"]
Nov 03 20:35:41.641565 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:41.641534    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:41.653398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:41.653370    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f" event={"ID":"6185bc78-db05-49ac-a333-426e95fcba93","Type":"ContainerStarted","Data":"76998c8226011841532d0aff179d573edf9c43419415d8cde8d89cc5ae460b3f"}
Nov 03 20:35:41.653539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:41.653476    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:41.654625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:41.654605    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq" event={"ID":"db8f8416-28df-478a-9197-b03d494ebaf9","Type":"ContainerStarted","Data":"520dd48fbec2f4875e111f75a42fa29686c7dc4618648644cb1a0c54a76149ef"}
Nov 03 20:35:41.856852 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:41.856789    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f" podStartSLOduration=1.981844627 podStartE2EDuration="6.856771881s" totalImagesPullingTime="4.394068163s" totalInitContainerRuntime="480.859091ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:41.854856779 +0000 UTC m=+334.834413622" watchObservedRunningTime="2026-11-03 20:35:41.856771881 +0000 UTC m=+334.836328721"
Nov 03 20:35:42.366314 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:42.366280    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:42.833614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:42.833542    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:43.151967 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:43.151934    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:43.352813 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:43.352759    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:43.826713 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:43.826683    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:44.043953 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:44.043918    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:45.206790 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:45.206761    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:47.265578 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:47.265542    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:47.815822 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:47.815790    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.227521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.227490    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.466001 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.465969    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"]
Nov 03 20:35:48.468883 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.468863    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:48.572015 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.571928    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 20:35:48.572015 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.571978    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 20:35:48.572191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.572018    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jmjmh\""
Nov 03 20:35:48.573108 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.573090    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"]
Nov 03 20:35:48.663592 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.663564    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pdz\" (UniqueName: \"kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz\") pod \"kuadrant-operator-controller-manager-678d87766f-twpbm\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:48.663731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.663630    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-twpbm\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:48.764629 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.764598    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pdz\" (UniqueName: \"kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz\") pod \"kuadrant-operator-controller-manager-678d87766f-twpbm\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:48.764760 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.764657    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-twpbm\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:48.765032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.765013    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-twpbm\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:48.823014 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:48.822945    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pdz\" (UniqueName: \"kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz\") pod \"kuadrant-operator-controller-manager-678d87766f-twpbm\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:49.080952 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:49.080875    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:49.539625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:49.539591    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"]
Nov 03 20:35:49.588371 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:35:49.588323    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a625db3_351f_421c_82f0_8850f8c3367c.slice/crio-c1787c4a69f65c695b756721a5824aa50679d06d5a9dfd5917a71581350f8da8 WatchSource:0}: Error finding container c1787c4a69f65c695b756721a5824aa50679d06d5a9dfd5917a71581350f8da8: Status 404 returned error can't find the container with id c1787c4a69f65c695b756721a5824aa50679d06d5a9dfd5917a71581350f8da8
Nov 03 20:35:49.693742 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:49.693705    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm" event={"ID":"4a625db3-351f-421c-82f0-8850f8c3367c","Type":"ContainerStarted","Data":"c1787c4a69f65c695b756721a5824aa50679d06d5a9dfd5917a71581350f8da8"}
Nov 03 20:35:52.661813 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:52.661783    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-zz68f"
Nov 03 20:35:54.774709 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:54.774671    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:54.984523 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:54.984490    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:55.721553 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:55.721519    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm" event={"ID":"4a625db3-351f-421c-82f0-8850f8c3367c","Type":"ContainerStarted","Data":"fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936"}
Nov 03 20:35:55.721739 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:55.721591    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:35:55.934616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:55.934275    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm" podStartSLOduration=2.589600707 podStartE2EDuration="7.934258497s" totalImagesPullingTime="5.34465779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:55.886692518 +0000 UTC m=+348.866249359" watchObservedRunningTime="2026-11-03 20:35:55.934258497 +0000 UTC m=+348.913815338"
Nov 03 20:35:55.935953 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:35:55.935926    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:35:58.331387 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.331346    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:58.565248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.565208    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:58.806913 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.806880    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"]
Nov 03 20:35:58.809956 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.809934    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:58.839561 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.839529    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/06588012-08b6-435b-8ecb-b5ed6d29f515-kube-api-access-88mtk\") pod \"cert-manager-webhook-5dbbbf75cc-g74vc\" (UID: \"06588012-08b6-435b-8ecb-b5ed6d29f515\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:58.839682 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.839561    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06588012-08b6-435b-8ecb-b5ed6d29f515-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g74vc\" (UID: \"06588012-08b6-435b-8ecb-b5ed6d29f515\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:58.909528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.909493    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s57nm\""
Nov 03 20:35:58.911292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.911267    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"]
Nov 03 20:35:58.915521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.915494    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 20:35:58.920058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.920039    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 20:35:58.940774 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.940747    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/06588012-08b6-435b-8ecb-b5ed6d29f515-kube-api-access-88mtk\") pod \"cert-manager-webhook-5dbbbf75cc-g74vc\" (UID: \"06588012-08b6-435b-8ecb-b5ed6d29f515\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:58.940874 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:58.940780    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06588012-08b6-435b-8ecb-b5ed6d29f515-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g74vc\" (UID: \"06588012-08b6-435b-8ecb-b5ed6d29f515\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:59.002643 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.002616    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06588012-08b6-435b-8ecb-b5ed6d29f515-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g74vc\" (UID: \"06588012-08b6-435b-8ecb-b5ed6d29f515\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:59.002892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.002862    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/06588012-08b6-435b-8ecb-b5ed6d29f515-kube-api-access-88mtk\") pod \"cert-manager-webhook-5dbbbf75cc-g74vc\" (UID: \"06588012-08b6-435b-8ecb-b5ed6d29f515\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:59.122813 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.122753    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:35:59.385088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.385010    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:59.491650 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.491621    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"]
Nov 03 20:35:59.494665 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:35:59.494634    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06588012_08b6_435b_8ecb_b5ed6d29f515.slice/crio-5f297151c3656fe0e7ae982bffa6fab48273568465986f50814e2a929716f7fb WatchSource:0}: Error finding container 5f297151c3656fe0e7ae982bffa6fab48273568465986f50814e2a929716f7fb: Status 404 returned error can't find the container with id 5f297151c3656fe0e7ae982bffa6fab48273568465986f50814e2a929716f7fb
Nov 03 20:35:59.587227 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.587198    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:59.740447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.740364    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc" event={"ID":"06588012-08b6-435b-8ecb-b5ed6d29f515","Type":"ContainerStarted","Data":"5f297151c3656fe0e7ae982bffa6fab48273568465986f50814e2a929716f7fb"}
Nov 03 20:35:59.777977 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.777940    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"]
Nov 03 20:35:59.780899 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.780882    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.807326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.807300    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:59.829524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.829504    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tfjxg\""
Nov 03 20:35:59.829614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.829547    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 20:35:59.829614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.829551    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 20:35:59.834456 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.834437    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"]
Nov 03 20:35:59.847499 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.847475    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pvh\" (UniqueName: \"kubernetes.io/projected/6218182d-10a0-4142-b413-46c95a9ecee8-kube-api-access-r7pvh\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.847593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.847513    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6218182d-10a0-4142-b413-46c95a9ecee8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.847593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.847549    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6218182d-10a0-4142-b413-46c95a9ecee8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.948863 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.948831    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6218182d-10a0-4142-b413-46c95a9ecee8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.948955 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.948885    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6218182d-10a0-4142-b413-46c95a9ecee8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.948955 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.948921    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pvh\" (UniqueName: \"kubernetes.io/projected/6218182d-10a0-4142-b413-46c95a9ecee8-kube-api-access-r7pvh\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.949455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.949434    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6218182d-10a0-4142-b413-46c95a9ecee8-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:35:59.951122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:35:59.951100    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6218182d-10a0-4142-b413-46c95a9ecee8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:36:00.006439 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:00.006385    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pvh\" (UniqueName: \"kubernetes.io/projected/6218182d-10a0-4142-b413-46c95a9ecee8-kube-api-access-r7pvh\") pod \"kuadrant-console-plugin-66c4fb575d-llhxk\" (UID: \"6218182d-10a0-4142-b413-46c95a9ecee8\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:36:00.016397 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:00.016364    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:00.092755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:00.092732    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"
Nov 03 20:36:00.217566 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:00.217540    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:00.465054 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:00.465015    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk"]
Nov 03 20:36:00.468225 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:36:00.468192    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6218182d_10a0_4142_b413_46c95a9ecee8.slice/crio-756ce9f653f3cdd2a72af11fe0c313c720176be8987388fb94068b9ebb57975c WatchSource:0}: Error finding container 756ce9f653f3cdd2a72af11fe0c313c720176be8987388fb94068b9ebb57975c: Status 404 returned error can't find the container with id 756ce9f653f3cdd2a72af11fe0c313c720176be8987388fb94068b9ebb57975c
Nov 03 20:36:00.746668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:00.746591    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk" event={"ID":"6218182d-10a0-4142-b413-46c95a9ecee8","Type":"ContainerStarted","Data":"756ce9f653f3cdd2a72af11fe0c313c720176be8987388fb94068b9ebb57975c"}
Nov 03 20:36:01.031601 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:01.031529    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.755538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:01.754764    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq" event={"ID":"db8f8416-28df-478a-9197-b03d494ebaf9","Type":"ContainerStarted","Data":"57cd019cf9162b260db8b0af2c1c7914a5715f5ebab448c270f681f147fa4b9f"}
Nov 03 20:36:01.874078 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:01.874012    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-cflnq" podStartSLOduration=2.155867873 podStartE2EDuration="22.873993019s" totalImagesPullingTime="20.718125146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:01.872546525 +0000 UTC m=+354.852103366" watchObservedRunningTime="2026-11-03 20:36:01.873993019 +0000 UTC m=+354.853549860"
Nov 03 20:36:03.994516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:03.994480    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:04.189636 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:04.189600    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:04.395132 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:04.395095    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:04.773406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:04.773373    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc" event={"ID":"06588012-08b6-435b-8ecb-b5ed6d29f515","Type":"ContainerStarted","Data":"0f2e2a3324a16b7e918ce0d8b3785fbe8013fd206b70d6b2bf6717ec5c846c6c"}
Nov 03 20:36:04.773652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:04.773473    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:36:04.882426 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:04.882370    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc" podStartSLOduration=2.108577168 podStartE2EDuration="6.882354523s" totalImagesPullingTime="4.773777355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:04.881578081 +0000 UTC m=+357.861134922" watchObservedRunningTime="2026-11-03 20:36:04.882354523 +0000 UTC m=+357.861911361"
Nov 03 20:36:06.727886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:06.727862    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 20:36:06.752189 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:06.752138    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:06.782850 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:06.782822    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk" event={"ID":"6218182d-10a0-4142-b413-46c95a9ecee8","Type":"ContainerStarted","Data":"0bfd782718f6fb06bb33437cd46b9f0ade989ea131c63c7d5492299d4fe4f355"}
Nov 03 20:36:06.972164 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:06.972134    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:07.049162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:07.049068    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-llhxk" podStartSLOduration=2.039271438 podStartE2EDuration="8.049055023s" totalImagesPullingTime="6.009783585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:07.047909967 +0000 UTC m=+360.027466807" watchObservedRunningTime="2026-11-03 20:36:07.049055023 +0000 UTC m=+360.028611855"
Nov 03 20:36:08.646637 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:08.646606    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:08.915228 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:08.915146    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:09.675442 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:09.675390    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:10.512102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.512069    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-8dd4r"]
Nov 03 20:36:10.515540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.515524    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.538410 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.538390    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwvl\" (UniqueName: \"kubernetes.io/projected/97878dfa-4e7e-480c-939f-4e2e2272c768-kube-api-access-fkwvl\") pod \"cert-manager-597b95cb84-8dd4r\" (UID: \"97878dfa-4e7e-480c-939f-4e2e2272c768\") " pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.538512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.538455    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97878dfa-4e7e-480c-939f-4e2e2272c768-bound-sa-token\") pod \"cert-manager-597b95cb84-8dd4r\" (UID: \"97878dfa-4e7e-480c-939f-4e2e2272c768\") " pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.610249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.610224    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qqcw7\""
Nov 03 20:36:10.611662 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.611642    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8dd4r"]
Nov 03 20:36:10.639126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.639103    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97878dfa-4e7e-480c-939f-4e2e2272c768-bound-sa-token\") pod \"cert-manager-597b95cb84-8dd4r\" (UID: \"97878dfa-4e7e-480c-939f-4e2e2272c768\") " pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.639211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.639149    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwvl\" (UniqueName: \"kubernetes.io/projected/97878dfa-4e7e-480c-939f-4e2e2272c768-kube-api-access-fkwvl\") pod \"cert-manager-597b95cb84-8dd4r\" (UID: \"97878dfa-4e7e-480c-939f-4e2e2272c768\") " pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.695990 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.695963    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwvl\" (UniqueName: \"kubernetes.io/projected/97878dfa-4e7e-480c-939f-4e2e2272c768-kube-api-access-fkwvl\") pod \"cert-manager-597b95cb84-8dd4r\" (UID: \"97878dfa-4e7e-480c-939f-4e2e2272c768\") " pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.696254 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.695987    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97878dfa-4e7e-480c-939f-4e2e2272c768-bound-sa-token\") pod \"cert-manager-597b95cb84-8dd4r\" (UID: \"97878dfa-4e7e-480c-939f-4e2e2272c768\") " pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:10.781806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.781727    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g74vc"
Nov 03 20:36:10.825924 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:10.825906    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8dd4r"
Nov 03 20:36:11.190022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:11.189993    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8dd4r"]
Nov 03 20:36:11.192563 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:36:11.192538    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97878dfa_4e7e_480c_939f_4e2e2272c768.slice/crio-dfc7caeb543c20f42f90b271db3afb5fd9205cfec48d3e266fd433fd037e9f9c WatchSource:0}: Error finding container dfc7caeb543c20f42f90b271db3afb5fd9205cfec48d3e266fd433fd037e9f9c: Status 404 returned error can't find the container with id dfc7caeb543c20f42f90b271db3afb5fd9205cfec48d3e266fd433fd037e9f9c
Nov 03 20:36:11.804325 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:11.804286    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8dd4r" event={"ID":"97878dfa-4e7e-480c-939f-4e2e2272c768","Type":"ContainerStarted","Data":"6e893975a40325bd602a8b79e58ce68885e6b24b12e1f802cfa6d4bfdc15e837"}
Nov 03 20:36:11.804812 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:11.804331    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8dd4r" event={"ID":"97878dfa-4e7e-480c-939f-4e2e2272c768","Type":"ContainerStarted","Data":"dfc7caeb543c20f42f90b271db3afb5fd9205cfec48d3e266fd433fd037e9f9c"}
Nov 03 20:36:11.915053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:11.915007    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-8dd4r" podStartSLOduration=1.914993374 podStartE2EDuration="1.914993374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:11.913563918 +0000 UTC m=+364.893120760" watchObservedRunningTime="2026-11-03 20:36:11.914993374 +0000 UTC m=+364.894550213"
Nov 03 20:36:14.729535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:14.729499    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:14.938003 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:14.937975    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:15.220507 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:15.220473    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.053574 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.053537    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.074427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.074397    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"]
Nov 03 20:36:17.077437 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.077414    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.127065 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.127038    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 20:36:17.127228 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.127207    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:36:17.127318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.127300    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ntmpg\""
Nov 03 20:36:17.127456 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.127436    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 20:36:17.173383 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.173360    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:36:17.175242 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.175219    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"]
Nov 03 20:36:17.191998 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.191978    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62eb7e47-ff1b-43e1-8752-c16af0654016-tls-certs\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.191998 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.192004    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62eb7e47-ff1b-43e1-8752-c16af0654016-tmp\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.192136 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.192032    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqfq\" (UniqueName: \"kubernetes.io/projected/62eb7e47-ff1b-43e1-8752-c16af0654016-kube-api-access-qxqfq\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.253877 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.253849    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.292934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.292905    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62eb7e47-ff1b-43e1-8752-c16af0654016-tls-certs\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.292934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.292936    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62eb7e47-ff1b-43e1-8752-c16af0654016-tmp\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.293085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.292958    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqfq\" (UniqueName: \"kubernetes.io/projected/62eb7e47-ff1b-43e1-8752-c16af0654016-kube-api-access-qxqfq\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.295121 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.295099    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62eb7e47-ff1b-43e1-8752-c16af0654016-tmp\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.295377 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.295361    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62eb7e47-ff1b-43e1-8752-c16af0654016-tls-certs\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.350138 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.350107    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqfq\" (UniqueName: \"kubernetes.io/projected/62eb7e47-ff1b-43e1-8752-c16af0654016-kube-api-access-qxqfq\") pod \"kube-auth-proxy-7ccb65679-h9rdf\" (UID: \"62eb7e47-ff1b-43e1-8752-c16af0654016\") " pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.389738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.389713    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"
Nov 03 20:36:17.466047 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.466022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.752544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.752520    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf"]
Nov 03 20:36:17.754934 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:36:17.754898    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eb7e47_ff1b_43e1_8752_c16af0654016.slice/crio-b72c31edf331b7b3ea56aa15efe64adacf600863aaf2854c2b71ca875c63b531 WatchSource:0}: Error finding container b72c31edf331b7b3ea56aa15efe64adacf600863aaf2854c2b71ca875c63b531: Status 404 returned error can't find the container with id b72c31edf331b7b3ea56aa15efe64adacf600863aaf2854c2b71ca875c63b531
Nov 03 20:36:17.831071 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.831044    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf" event={"ID":"62eb7e47-ff1b-43e1-8752-c16af0654016","Type":"ContainerStarted","Data":"b72c31edf331b7b3ea56aa15efe64adacf600863aaf2854c2b71ca875c63b531"}
Nov 03 20:36:17.846814 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:17.846788    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:18.135581 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:18.135551    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:19.513352 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:19.513301    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:19.764097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:19.764013    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:19.968266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:19.968238    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:21.093689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:21.093654    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:21.294256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:21.294216    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:21.850265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:21.850236    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf" event={"ID":"62eb7e47-ff1b-43e1-8752-c16af0654016","Type":"ContainerStarted","Data":"d52e88977db1b549b2cf5c3207fb033b10876e230b8174e7fa6ff25f1e09644f"}
Nov 03 20:36:21.959058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:21.959006    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ccb65679-h9rdf" podStartSLOduration=1.132634485 podStartE2EDuration="4.958990944s" totalImagesPullingTime="3.826356459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:21.957648207 +0000 UTC m=+374.937205047" watchObservedRunningTime="2026-11-03 20:36:21.958990944 +0000 UTC m=+374.938547787"
Nov 03 20:36:22.699181 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.699150    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:36:22.701876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.701859    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.742580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.742555    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec3e62c6-7c98-4e0e-ad88-4653f9a1982e-config-file\") pod \"limitador-limitador-7c9b87d574-j6xpz\" (UID: \"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.742678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.742603    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjn9\" (UniqueName: \"kubernetes.io/projected/ec3e62c6-7c98-4e0e-ad88-4653f9a1982e-kube-api-access-npjn9\") pod \"limitador-limitador-7c9b87d574-j6xpz\" (UID: \"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.761152 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.761128    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:36:22.768089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.768073    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 20:36:22.843378 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.843328    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec3e62c6-7c98-4e0e-ad88-4653f9a1982e-config-file\") pod \"limitador-limitador-7c9b87d574-j6xpz\" (UID: \"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.843490 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.843406    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjn9\" (UniqueName: \"kubernetes.io/projected/ec3e62c6-7c98-4e0e-ad88-4653f9a1982e-kube-api-access-npjn9\") pod \"limitador-limitador-7c9b87d574-j6xpz\" (UID: \"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.843940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.843921    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec3e62c6-7c98-4e0e-ad88-4653f9a1982e-config-file\") pod \"limitador-limitador-7c9b87d574-j6xpz\" (UID: \"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.899969 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.899946    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjn9\" (UniqueName: \"kubernetes.io/projected/ec3e62c6-7c98-4e0e-ad88-4653f9a1982e-kube-api-access-npjn9\") pod \"limitador-limitador-7c9b87d574-j6xpz\" (UID: \"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:22.997787 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:22.997724    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:36:23.013438 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:23.013414    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:23.373609 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:23.373578    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:36:23.376390 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:36:23.376359    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3e62c6_7c98_4e0e_ad88_4653f9a1982e.slice/crio-1aeaa1baa056891863aae8cafbda9531a18bf71a77a2dd935941983e10c8a96a WatchSource:0}: Error finding container 1aeaa1baa056891863aae8cafbda9531a18bf71a77a2dd935941983e10c8a96a: Status 404 returned error can't find the container with id 1aeaa1baa056891863aae8cafbda9531a18bf71a77a2dd935941983e10c8a96a
Nov 03 20:36:23.860773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:23.860740    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz" event={"ID":"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e","Type":"ContainerStarted","Data":"1aeaa1baa056891863aae8cafbda9531a18bf71a77a2dd935941983e10c8a96a"}
Nov 03 20:36:25.197041 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:25.197007    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:25.934952 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:36:25.934752    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:36:26.877319 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:26.877287    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz" event={"ID":"ec3e62c6-7c98-4e0e-ad88-4653f9a1982e","Type":"ContainerStarted","Data":"78926e3f49c0f26214479e553133bf8037cedda9a31ac4f8d493cd6987761d5a"}
Nov 03 20:36:26.877723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:26.877399    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:26.986322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:26.986271    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz" podStartSLOduration=2.428895452 podStartE2EDuration="4.98625547s" totalImagesPullingTime="2.557360018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:26.985064826 +0000 UTC m=+379.964621666" watchObservedRunningTime="2026-11-03 20:36:26.98625547 +0000 UTC m=+379.965812309"
Nov 03 20:36:30.798943 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:30.798890    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:30.993015 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:30.992980    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:37.882256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:37.882226    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"
Nov 03 20:36:42.148668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:42.148635    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:42.351462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:42.351398    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:42.963665 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:42.963632    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.463848 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:46.463767    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.677473 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:46.677404    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.881942 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:36:46.881916    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:55.932910 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:36:55.932875    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:37:04.476614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:04.476577    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:04.672950 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:04.672920    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:04.871741 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:04.871709    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:05.073704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:05.073665    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:09.581718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:09.581680    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:09.793794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:09.793767    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:12.546269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:12.546232    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:12.818265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:12.818183    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:15.265822 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:15.265791    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:25.932473 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:37:25.932429    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:37:32.015274 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:32.015233    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:32.230520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:32.230477    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:37.373520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:37.373483    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:37.604886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:37.604831    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:37.843436 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:37.843403    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.089241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.089210    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.168407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.168316    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bgkgw"]
Nov 03 20:37:38.172144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.172127    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.231053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.231028    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 20:37:38.235846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.235790    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 20:37:38.257991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.257958    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq964\" (UniqueName: \"kubernetes.io/projected/4a1004ed-d52d-4a18-a74a-590f023c59f8-kube-api-access-lq964\") pod \"kserve-controller-manager-776cc66d47-bgkgw\" (UID: \"4a1004ed-d52d-4a18-a74a-590f023c59f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.258119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.258002    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a1004ed-d52d-4a18-a74a-590f023c59f8-cert\") pod \"kserve-controller-manager-776cc66d47-bgkgw\" (UID: \"4a1004ed-d52d-4a18-a74a-590f023c59f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.268242 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.268220    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 20:37:38.268299 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.268220    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-thq28\""
Nov 03 20:37:38.269872 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.269853    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bgkgw"]
Nov 03 20:37:38.290511 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.290486    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"]
Nov 03 20:37:38.293828 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.293812    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.344154 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.344126    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 20:37:38.344258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.344175    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4rfvp\""
Nov 03 20:37:38.353050 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.353022    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"]
Nov 03 20:37:38.355953 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.355928    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.359068 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.359043    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-86th5\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.359171 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.359087    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq964\" (UniqueName: \"kubernetes.io/projected/4a1004ed-d52d-4a18-a74a-590f023c59f8-kube-api-access-lq964\") pod \"kserve-controller-manager-776cc66d47-bgkgw\" (UID: \"4a1004ed-d52d-4a18-a74a-590f023c59f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.359171 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.359105    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w56\" (UniqueName: \"kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56\") pod \"llmisvc-controller-manager-869d5dcc5d-86th5\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.359290 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.359215    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a1004ed-d52d-4a18-a74a-590f023c59f8-cert\") pod \"kserve-controller-manager-776cc66d47-bgkgw\" (UID: \"4a1004ed-d52d-4a18-a74a-590f023c59f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.361559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.361538    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a1004ed-d52d-4a18-a74a-590f023c59f8-cert\") pod \"kserve-controller-manager-776cc66d47-bgkgw\" (UID: \"4a1004ed-d52d-4a18-a74a-590f023c59f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.410250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.410225    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kzmvk"]
Nov 03 20:37:38.413432 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.413418    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.419655 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.419612    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq964\" (UniqueName: \"kubernetes.io/projected/4a1004ed-d52d-4a18-a74a-590f023c59f8-kube-api-access-lq964\") pod \"kserve-controller-manager-776cc66d47-bgkgw\" (UID: \"4a1004ed-d52d-4a18-a74a-590f023c59f8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.459863 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.459836    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-86th5\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.459958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.459877    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w56\" (UniqueName: \"kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56\") pod \"llmisvc-controller-manager-869d5dcc5d-86th5\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.462096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.462076    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-86th5\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.462678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.462663    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dlxc6\""
Nov 03 20:37:38.465544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.465531    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 20:37:38.469040 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.468995    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kzmvk"]
Nov 03 20:37:38.484996 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.484977    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:38.515500 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.515473    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-tdb2j"]
Nov 03 20:37:38.519929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.519908    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.526572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.526548    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w56\" (UniqueName: \"kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56\") pod \"llmisvc-controller-manager-869d5dcc5d-86th5\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.560540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.560510    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4j6v\" (UniqueName: \"kubernetes.io/projected/de74614d-9ad6-4417-a716-0fc1375715b6-kube-api-access-b4j6v\") pod \"model-serving-api-75bd89fdfc-kzmvk\" (UID: \"de74614d-9ad6-4417-a716-0fc1375715b6\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.560666 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.560573    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de74614d-9ad6-4417-a716-0fc1375715b6-tls-certs\") pod \"model-serving-api-75bd89fdfc-kzmvk\" (UID: \"de74614d-9ad6-4417-a716-0fc1375715b6\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.572294 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.572268    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 20:37:38.572440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.572347    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j2nxk\""
Nov 03 20:37:38.574434 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.574412    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-tdb2j"]
Nov 03 20:37:38.574543 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.574502    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 20:37:38.606501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.606479    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:38.662412 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.662380    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de74614d-9ad6-4417-a716-0fc1375715b6-tls-certs\") pod \"model-serving-api-75bd89fdfc-kzmvk\" (UID: \"de74614d-9ad6-4417-a716-0fc1375715b6\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.662548 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.662427    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb6f8913-899f-4e7b-874a-7325937d52a9-cert\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.662548 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.662533    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzj5n\" (UniqueName: \"kubernetes.io/projected/eb6f8913-899f-4e7b-874a-7325937d52a9-kube-api-access-lzj5n\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.662668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.662574    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f8913-899f-4e7b-874a-7325937d52a9-metrics-certs\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.662668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.662640    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4j6v\" (UniqueName: \"kubernetes.io/projected/de74614d-9ad6-4417-a716-0fc1375715b6-kube-api-access-b4j6v\") pod \"model-serving-api-75bd89fdfc-kzmvk\" (UID: \"de74614d-9ad6-4417-a716-0fc1375715b6\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.665888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.665865    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de74614d-9ad6-4417-a716-0fc1375715b6-tls-certs\") pod \"model-serving-api-75bd89fdfc-kzmvk\" (UID: \"de74614d-9ad6-4417-a716-0fc1375715b6\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.719875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.719812    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4j6v\" (UniqueName: \"kubernetes.io/projected/de74614d-9ad6-4417-a716-0fc1375715b6-kube-api-access-b4j6v\") pod \"model-serving-api-75bd89fdfc-kzmvk\" (UID: \"de74614d-9ad6-4417-a716-0fc1375715b6\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.733340 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.733323    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:38.763602 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.763569    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f8913-899f-4e7b-874a-7325937d52a9-metrics-certs\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.763725 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.763712    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb6f8913-899f-4e7b-874a-7325937d52a9-cert\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.763786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.763774    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzj5n\" (UniqueName: \"kubernetes.io/projected/eb6f8913-899f-4e7b-874a-7325937d52a9-kube-api-access-lzj5n\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.766014 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.765994    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb6f8913-899f-4e7b-874a-7325937d52a9-cert\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.766119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.766043    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6f8913-899f-4e7b-874a-7325937d52a9-metrics-certs\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.821720 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.821690    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzj5n\" (UniqueName: \"kubernetes.io/projected/eb6f8913-899f-4e7b-874a-7325937d52a9-kube-api-access-lzj5n\") pod \"odh-model-controller-769875db56-tdb2j\" (UID: \"eb6f8913-899f-4e7b-874a-7325937d52a9\") " pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.836474 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.836452    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:38.870972 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.870939    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bgkgw"]
Nov 03 20:37:38.968973 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:38.968904    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"]
Nov 03 20:37:38.971502 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:37:38.971445    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc440662b_83f3_453a_a167_8a431d01405c.slice/crio-7651aa787ec2fc931ee4310c475af4cdbaf09c9cdea7dd279ccfa48bc1d9edae WatchSource:0}: Error finding container 7651aa787ec2fc931ee4310c475af4cdbaf09c9cdea7dd279ccfa48bc1d9edae: Status 404 returned error can't find the container with id 7651aa787ec2fc931ee4310c475af4cdbaf09c9cdea7dd279ccfa48bc1d9edae
Nov 03 20:37:39.096345 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.096309    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kzmvk"]
Nov 03 20:37:39.098771 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:37:39.098743    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde74614d_9ad6_4417_a716_0fc1375715b6.slice/crio-abee9225898ba2fb158380b164efbe88007bcba3a8d31878ecce3cd765563550 WatchSource:0}: Error finding container abee9225898ba2fb158380b164efbe88007bcba3a8d31878ecce3cd765563550: Status 404 returned error can't find the container with id abee9225898ba2fb158380b164efbe88007bcba3a8d31878ecce3cd765563550
Nov 03 20:37:39.202347 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.202306    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk" event={"ID":"de74614d-9ad6-4417-a716-0fc1375715b6","Type":"ContainerStarted","Data":"abee9225898ba2fb158380b164efbe88007bcba3a8d31878ecce3cd765563550"}
Nov 03 20:37:39.203515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.203489    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5" event={"ID":"c440662b-83f3-453a-a167-8a431d01405c","Type":"ContainerStarted","Data":"7651aa787ec2fc931ee4310c475af4cdbaf09c9cdea7dd279ccfa48bc1d9edae"}
Nov 03 20:37:39.204615 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.204585    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw" event={"ID":"4a1004ed-d52d-4a18-a74a-590f023c59f8","Type":"ContainerStarted","Data":"6d03ca5201356b382df7b4482a7f9b476a37c225f951d367b57073abcd001acc"}
Nov 03 20:37:39.211729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.211706    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-tdb2j"]
Nov 03 20:37:39.214806 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:37:39.214782    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6f8913_899f_4e7b_874a_7325937d52a9.slice/crio-db322f373077a7ca8ba11ecf22dc1bc0ee5ad74e109235fd6f1c0c896b172abd WatchSource:0}: Error finding container db322f373077a7ca8ba11ecf22dc1bc0ee5ad74e109235fd6f1c0c896b172abd: Status 404 returned error can't find the container with id db322f373077a7ca8ba11ecf22dc1bc0ee5ad74e109235fd6f1c0c896b172abd
Nov 03 20:37:39.601024 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.600991    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:39.911671 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:39.911540    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.161943 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:40.161856    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.214092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:40.214049    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-tdb2j" event={"ID":"eb6f8913-899f-4e7b-874a-7325937d52a9","Type":"ContainerStarted","Data":"db322f373077a7ca8ba11ecf22dc1bc0ee5ad74e109235fd6f1c0c896b172abd"}
Nov 03 20:37:42.182205 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:42.182135    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:42.229761 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:42.229716    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk" event={"ID":"de74614d-9ad6-4417-a716-0fc1375715b6","Type":"ContainerStarted","Data":"1c382ffec7f9cece562243e2903cdd24e26cb14ee90debb79b5c387c601ba3e7"}
Nov 03 20:37:42.230039 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:42.229996    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:42.342572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:42.342512    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk" podStartSLOduration=1.414986226 podStartE2EDuration="4.342497351s" totalImagesPullingTime="2.927511125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:42.342079311 +0000 UTC m=+455.321636153" watchObservedRunningTime="2026-11-03 20:37:42.342497351 +0000 UTC m=+455.322054191"
Nov 03 20:37:42.389604 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:42.389564    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:42.591691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:42.591650    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:43.163064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.163030    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"]
Nov 03 20:37:43.166915 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.166888    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.236979 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.236942    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5" event={"ID":"c440662b-83f3-453a-a167-8a431d01405c","Type":"ContainerStarted","Data":"50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409"}
Nov 03 20:37:43.262817 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.262719    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\""
Nov 03 20:37:43.262817 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.262743    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 20:37:43.263076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.262905    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:37:43.263076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.263002    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:37:43.263831 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.263810    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:37:43.265128 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.265108    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"]
Nov 03 20:37:43.307347 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.307316    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.307477 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.307376    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.307522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.307489    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdl6c\" (UniqueName: \"kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.307557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.307538    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.407995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.407963    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.408155 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.408026    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.408155 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.408046    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.408155 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.408091    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdl6c\" (UniqueName: \"kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.408918 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.408888    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.429187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.429124    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.429315 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.429183    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.437544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.437471    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5" podStartSLOduration=1.813925018 podStartE2EDuration="5.437455438s" totalImagesPullingTime="3.62353042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:43.436525473 +0000 UTC m=+456.416082310" watchObservedRunningTime="2026-11-03 20:37:43.437455438 +0000 UTC m=+456.417012280"
Nov 03 20:37:43.465620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.465594    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdl6c\" (UniqueName: \"kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c\") pod \"maas-api-6d669c7fcd-gchsf\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.482432 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.482174    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:43.858873 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:43.858848    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"]
Nov 03 20:37:43.861346 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:37:43.861307    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6125098_5c55_4bfe_8a81_489eb496981c.slice/crio-55a6472fbb76366c16514b3d2ca817763440ff475ff4ea3d5ef6dfaa4b60a4f7 WatchSource:0}: Error finding container 55a6472fbb76366c16514b3d2ca817763440ff475ff4ea3d5ef6dfaa4b60a4f7: Status 404 returned error can't find the container with id 55a6472fbb76366c16514b3d2ca817763440ff475ff4ea3d5ef6dfaa4b60a4f7
Nov 03 20:37:44.244770 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:44.244688    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-tdb2j" event={"ID":"eb6f8913-899f-4e7b-874a-7325937d52a9","Type":"ContainerStarted","Data":"76345906e97205b8419c9433e4bda5158a4b74cb2b1f8228770e9502dc4b96b1"}
Nov 03 20:37:44.245159 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:44.244866    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:44.246381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:44.246355    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf" event={"ID":"b6125098-5c55-4bfe-8a81-489eb496981c","Type":"ContainerStarted","Data":"55a6472fbb76366c16514b3d2ca817763440ff475ff4ea3d5ef6dfaa4b60a4f7"}
Nov 03 20:37:44.246615 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:44.246591    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:37:44.360535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:44.360484    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-tdb2j" podStartSLOduration=1.9603262 podStartE2EDuration="6.360472051s" totalImagesPullingTime="4.400145851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:44.36039356 +0000 UTC m=+457.339950402" watchObservedRunningTime="2026-11-03 20:37:44.360472051 +0000 UTC m=+457.340028890"
Nov 03 20:37:44.857951 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:44.857918    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.102293 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.102261    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.257069 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.257036    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw" event={"ID":"4a1004ed-d52d-4a18-a74a-590f023c59f8","Type":"ContainerStarted","Data":"5b15250481fe57e56f8250460aa1b75d86af7384488fc237eb37626cebf61e8e"}
Nov 03 20:37:45.257435 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.257098    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:37:45.305409 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.305373    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.365069 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.365023    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw" podStartSLOduration=1.123762958 podStartE2EDuration="7.365009236s" totalImagesPullingTime="6.241246278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:45.364236075 +0000 UTC m=+458.343792916" watchObservedRunningTime="2026-11-03 20:37:45.365009236 +0000 UTC m=+458.344566076"
Nov 03 20:37:45.513360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.513239    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.723216 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.723182    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.941351 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:45.941303    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:46.203603 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:46.203523    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:48.280891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:48.280856    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf" event={"ID":"b6125098-5c55-4bfe-8a81-489eb496981c","Type":"ContainerStarted","Data":"56ca853216dbad2be34fe54bc5fd55de7778cd334a44f5822448d62a5c54e896"}
Nov 03 20:37:48.281259 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:48.280934    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:48.394368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:48.394302    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf" podStartSLOduration=1.551770801 podStartE2EDuration="5.394283989s" totalImagesPullingTime="3.842513188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:48.393044747 +0000 UTC m=+461.372601589" watchObservedRunningTime="2026-11-03 20:37:48.394283989 +0000 UTC m=+461.373840828"
Nov 03 20:37:49.288019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:49.287981    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:37:53.240839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:53.240801    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvk"
Nov 03 20:37:54.291527 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:54.291496    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:37:54.916400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:54.916363    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:55.238126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:55.238038    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:55.257254 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:37:55.257233    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-tdb2j"
Nov 03 20:37:55.932931 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:37:55.932900    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:38:06.240405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.240373    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"]
Nov 03 20:38:06.240943 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.240608    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf" podUID="b6125098-5c55-4bfe-8a81-489eb496981c" containerName="maas-api" containerID="cri-o://56ca853216dbad2be34fe54bc5fd55de7778cd334a44f5822448d62a5c54e896" gracePeriod=30
Nov 03 20:38:06.362625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.362595    5318 generic.go:356] "Generic (PLEG): container finished" podID="b6125098-5c55-4bfe-8a81-489eb496981c" containerID="56ca853216dbad2be34fe54bc5fd55de7778cd334a44f5822448d62a5c54e896" exitCode=0
Nov 03 20:38:06.362744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.362662    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf" event={"ID":"b6125098-5c55-4bfe-8a81-489eb496981c","Type":"ContainerDied","Data":"56ca853216dbad2be34fe54bc5fd55de7778cd334a44f5822448d62a5c54e896"}
Nov 03 20:38:06.528380 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.528326    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:38:06.624557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.624529    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs\") pod \"b6125098-5c55-4bfe-8a81-489eb496981c\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") "
Nov 03 20:38:06.624679 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.624563    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls\") pod \"b6125098-5c55-4bfe-8a81-489eb496981c\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") "
Nov 03 20:38:06.624679 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.624600    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca\") pod \"b6125098-5c55-4bfe-8a81-489eb496981c\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") "
Nov 03 20:38:06.624679 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.624676    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c\" (UniqueName: \"kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c\") pod \"b6125098-5c55-4bfe-8a81-489eb496981c\" (UID: \"b6125098-5c55-4bfe-8a81-489eb496981c\") "
Nov 03 20:38:06.624991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.624965    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca" pod "b6125098-5c55-4bfe-8a81-489eb496981c" (UID: "b6125098-5c55-4bfe-8a81-489eb496981c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:06.626763 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.626741    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs" pod "b6125098-5c55-4bfe-8a81-489eb496981c" (UID: "b6125098-5c55-4bfe-8a81-489eb496981c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:06.626877 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.626860    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c" pod "b6125098-5c55-4bfe-8a81-489eb496981c" (UID: "b6125098-5c55-4bfe-8a81-489eb496981c"). InnerVolumeSpecName "kube-api-access-bdl6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:06.627402 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.627383    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls" pod "b6125098-5c55-4bfe-8a81-489eb496981c" (UID: "b6125098-5c55-4bfe-8a81-489eb496981c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:06.725562 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.725535    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6125098-5c55-4bfe-8a81-489eb496981c-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:06.725562 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.725559    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdl6c\" (UniqueName: \"kubernetes.io/projected/b6125098-5c55-4bfe-8a81-489eb496981c-kube-api-access-bdl6c\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:06.725723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.725569    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:06.725723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:06.725580    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6125098-5c55-4bfe-8a81-489eb496981c-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:07.368139 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:07.368113    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"
Nov 03 20:38:07.368564 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:07.368115    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf" event={"ID":"b6125098-5c55-4bfe-8a81-489eb496981c","Type":"ContainerDied","Data":"55a6472fbb76366c16514b3d2ca817763440ff475ff4ea3d5ef6dfaa4b60a4f7"}
Nov 03 20:38:07.368564 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:07.368225    5318 scope.go:122] "RemoveContainer" containerID="56ca853216dbad2be34fe54bc5fd55de7778cd334a44f5822448d62a5c54e896"
Nov 03 20:38:07.525463 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:07.525411    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"]
Nov 03 20:38:07.571182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:07.571160    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-gchsf"]
Nov 03 20:38:09.253511 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:09.253477    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6125098-5c55-4bfe-8a81-489eb496981c" path="/var/lib/kubelet/pods/b6125098-5c55-4bfe-8a81-489eb496981c/volumes"
Nov 03 20:38:12.176358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:12.176298    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:15.256404 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:15.256373    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:38:15.610276 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:15.610245    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:16.274820 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:16.274793    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-bgkgw"
Nov 03 20:38:25.932663 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:38:25.932624    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:38:43.441460 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:43.441403    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:43.639048 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:43.639015    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:44.447055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.447018    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"]
Nov 03 20:38:44.447587 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.447567    5318 cpu_manager.go:414] "containerMap: removing container" podUID="b6125098-5c55-4bfe-8a81-489eb496981c" containerName="maas-api"
Nov 03 20:38:44.447664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.447589    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6125098-5c55-4bfe-8a81-489eb496981c" containerName="maas-api"
Nov 03 20:38:44.447724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.447705    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6125098-5c55-4bfe-8a81-489eb496981c" containerName="maas-api"
Nov 03 20:38:44.450204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.450183    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.505745 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.505719    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"]
Nov 03 20:38:44.539329 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.539304    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e65d8a64-1bbf-4e99-bed9-83307bcb8da6-cert\") pod \"llmisvc-controller-manager-6466c554d8-5swbs\" (UID: \"e65d8a64-1bbf-4e99-bed9-83307bcb8da6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.539475 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.539358    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5xd\" (UniqueName: \"kubernetes.io/projected/e65d8a64-1bbf-4e99-bed9-83307bcb8da6-kube-api-access-ns5xd\") pod \"llmisvc-controller-manager-6466c554d8-5swbs\" (UID: \"e65d8a64-1bbf-4e99-bed9-83307bcb8da6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.640719 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.640688    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e65d8a64-1bbf-4e99-bed9-83307bcb8da6-cert\") pod \"llmisvc-controller-manager-6466c554d8-5swbs\" (UID: \"e65d8a64-1bbf-4e99-bed9-83307bcb8da6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.640862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.640729    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5xd\" (UniqueName: \"kubernetes.io/projected/e65d8a64-1bbf-4e99-bed9-83307bcb8da6-kube-api-access-ns5xd\") pod \"llmisvc-controller-manager-6466c554d8-5swbs\" (UID: \"e65d8a64-1bbf-4e99-bed9-83307bcb8da6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.643157 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.643132    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e65d8a64-1bbf-4e99-bed9-83307bcb8da6-cert\") pod \"llmisvc-controller-manager-6466c554d8-5swbs\" (UID: \"e65d8a64-1bbf-4e99-bed9-83307bcb8da6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.697909 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.697851    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5xd\" (UniqueName: \"kubernetes.io/projected/e65d8a64-1bbf-4e99-bed9-83307bcb8da6-kube-api-access-ns5xd\") pod \"llmisvc-controller-manager-6466c554d8-5swbs\" (UID: \"e65d8a64-1bbf-4e99-bed9-83307bcb8da6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:44.761930 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:44.761907    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:45.125831 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:45.125806    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"]
Nov 03 20:38:45.128309 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:38:45.128279    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65d8a64_1bbf_4e99_bed9_83307bcb8da6.slice/crio-fb4e0166815a0fe68512da79e6bf6cfae9ddeb35fcb6465c9e5cd97589568a32 WatchSource:0}: Error finding container fb4e0166815a0fe68512da79e6bf6cfae9ddeb35fcb6465c9e5cd97589568a32: Status 404 returned error can't find the container with id fb4e0166815a0fe68512da79e6bf6cfae9ddeb35fcb6465c9e5cd97589568a32
Nov 03 20:38:45.557166 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:45.557088    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs" event={"ID":"e65d8a64-1bbf-4e99-bed9-83307bcb8da6","Type":"ContainerStarted","Data":"fb4e0166815a0fe68512da79e6bf6cfae9ddeb35fcb6465c9e5cd97589568a32"}
Nov 03 20:38:46.562983 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:46.562950    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs" event={"ID":"e65d8a64-1bbf-4e99-bed9-83307bcb8da6","Type":"ContainerStarted","Data":"64d4371e1e81438b0d9b8d9668848d39e5c8755e50513a7804ab542c99f03f4b"}
Nov 03 20:38:46.563412 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:46.563069    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:38:46.672842 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:38:46.672793    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs" podStartSLOduration=1.948588228 podStartE2EDuration="2.672780255s" totalImagesPullingTime="724.192027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:46.671541153 +0000 UTC m=+519.651097993" watchObservedRunningTime="2026-11-03 20:38:46.672780255 +0000 UTC m=+519.652337095"
Nov 03 20:38:55.932254 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:38:55.932218    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:39:15.945301 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:15.945226    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:17.569895 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:17.569862    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5swbs"
Nov 03 20:39:17.711066 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:17.711031    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"]
Nov 03 20:39:17.711305 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:17.711277    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5" podUID="c440662b-83f3-453a-a167-8a431d01405c" containerName="manager" containerID="cri-o://50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409" gracePeriod=30
Nov 03 20:39:17.952670 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:17.952644    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:39:18.026713 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.026689    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56\" (UniqueName: \"kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56\") pod \"c440662b-83f3-453a-a167-8a431d01405c\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") "
Nov 03 20:39:18.026857 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.026792    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert\" (UniqueName: \"kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert\") pod \"c440662b-83f3-453a-a167-8a431d01405c\" (UID: \"c440662b-83f3-453a-a167-8a431d01405c\") "
Nov 03 20:39:18.028771 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.028737    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56" pod "c440662b-83f3-453a-a167-8a431d01405c" (UID: "c440662b-83f3-453a-a167-8a431d01405c"). InnerVolumeSpecName "kube-api-access-w7w56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:18.028771 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.028756    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert" pod "c440662b-83f3-453a-a167-8a431d01405c" (UID: "c440662b-83f3-453a-a167-8a431d01405c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:18.128005 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.127977    5318 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c440662b-83f3-453a-a167-8a431d01405c-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.128005 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.128004    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7w56\" (UniqueName: \"kubernetes.io/projected/c440662b-83f3-453a-a167-8a431d01405c-kube-api-access-w7w56\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.539569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.539497    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.711933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.711897    5318 generic.go:356] "Generic (PLEG): container finished" podID="c440662b-83f3-453a-a167-8a431d01405c" containerID="50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409" exitCode=0
Nov 03 20:39:18.712398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.711945    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"
Nov 03 20:39:18.712398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.711962    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5" event={"ID":"c440662b-83f3-453a-a167-8a431d01405c","Type":"ContainerDied","Data":"50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409"}
Nov 03 20:39:18.712398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.712016    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5" event={"ID":"c440662b-83f3-453a-a167-8a431d01405c","Type":"ContainerDied","Data":"7651aa787ec2fc931ee4310c475af4cdbaf09c9cdea7dd279ccfa48bc1d9edae"}
Nov 03 20:39:18.712398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.712047    5318 scope.go:122] "RemoveContainer" containerID="50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409"
Nov 03 20:39:18.721470 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.721453    5318 scope.go:122] "RemoveContainer" containerID="50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409"
Nov 03 20:39:18.721725 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:39:18.721708    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409\": container with ID starting with 50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409 not found: ID does not exist" containerID="50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409"
Nov 03 20:39:18.721769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.721734    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409"} err="failed to get container status \"50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409\": rpc error: code = NotFound desc = could not find container \"50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409\": container with ID starting with 50b3ac2730044210da01a61c6fba6e99f1c9207525421ab606ff259d8bb52409 not found: ID does not exist"
Nov 03 20:39:18.758257 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.758234    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.885986 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.885956    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"]
Nov 03 20:39:18.891691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:18.891665    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-86th5"]
Nov 03 20:39:19.253884 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:19.253813    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c440662b-83f3-453a-a167-8a431d01405c" path="/var/lib/kubelet/pods/c440662b-83f3-453a-a167-8a431d01405c/volumes"
Nov 03 20:39:25.252297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:25.252252    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:25.457580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:25.457538    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:25.932643 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:39:25.932607    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:39:43.657590 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:43.657556    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:47.219018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:47.218979    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:39:47.765563 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:47.765524    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:51.004762 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:51.004719    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:54.326859 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:54.326827    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:55.932977 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:39:55.932944    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:39:58.876197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:39:58.876163    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:02.451970 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:02.451935    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:40:04.598333 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:04.598284    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:07.374026 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:07.373996    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:40:07.377691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:07.377673    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:40:09.432586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:09.432538    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:13.368792 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:13.368760    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:14.508082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:14.508055    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:14.759652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:14.759580    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:16.250450 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:16.250414    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:20.383823 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:20.383785    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:40:21.287989 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:21.287954    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:22.084986 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:22.084949    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:25.067992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.067960    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"]
Nov 03 20:40:25.068414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.068323    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c440662b-83f3-453a-a167-8a431d01405c" containerName="manager"
Nov 03 20:40:25.068414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.068348    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c440662b-83f3-453a-a167-8a431d01405c" containerName="manager"
Nov 03 20:40:25.068509 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.068437    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c440662b-83f3-453a-a167-8a431d01405c" containerName="manager"
Nov 03 20:40:25.070670 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.070654    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.167891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.167861    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"]
Nov 03 20:40:25.199142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199102    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199154    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199228    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199298    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199495 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199379    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpk5h\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199495 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199469    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199601 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199509    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199599    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.199704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.199666    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.211794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.211768    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:40:25.211794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.211769    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\""
Nov 03 20:40:25.300308 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300276    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300513 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300327    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300513 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300390    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300565    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300621    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300654    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300893 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300695    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300893 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300729    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.300893 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300753    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpk5h\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.301060 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.300974    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.301693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.301418    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.301693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.301458    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.301693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.301536    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.301693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.301579    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.303174 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.303147    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.304023 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.304003    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.357958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.357932    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.358064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.358049    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpk5h\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.384841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.384817    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:25.799747 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:25.799718    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"]
Nov 03 20:40:25.801782 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:40:25.801754    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe170fd_b1c2_413a_b80d_59e406f5e403.slice/crio-18c85c3860a33c07863fe0dcadac2d15e43e93c2ed031a331f65b4f51098bd4b WatchSource:0}: Error finding container 18c85c3860a33c07863fe0dcadac2d15e43e93c2ed031a331f65b4f51098bd4b: Status 404 returned error can't find the container with id 18c85c3860a33c07863fe0dcadac2d15e43e93c2ed031a331f65b4f51098bd4b
Nov 03 20:40:25.933055 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:40:25.933022    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:40:26.033633 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:26.033593    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" event={"ID":"afe170fd-b1c2-413a-b80d-59e406f5e403","Type":"ContainerStarted","Data":"18c85c3860a33c07863fe0dcadac2d15e43e93c2ed031a331f65b4f51098bd4b"}
Nov 03 20:40:29.334479 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:29.334441    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:40:29.334776 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:29.334520    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:40:29.334776 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:29.334548    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:40:30.040352 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.040309    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:30.057172 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.057142    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" event={"ID":"afe170fd-b1c2-413a-b80d-59e406f5e403","Type":"ContainerStarted","Data":"9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f"}
Nov 03 20:40:30.217689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.217633    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podStartSLOduration=1.687768926 podStartE2EDuration="5.217616387s" totalImagesPullingTime="3.529847461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:30.216583697 +0000 UTC m=+623.196140536" watchObservedRunningTime="2026-11-03 20:40:30.217616387 +0000 UTC m=+623.197173226"
Nov 03 20:40:30.242143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.242107    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:30.385743 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.385665    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:30.390538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.390509    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:30.390688 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:30.390563    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:31.386360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:31.385725    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:31.386360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:31.385802    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:32.385736 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:32.385696    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:32.386021 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:32.385992    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:33.391478 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:33.391440    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:33.392053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:33.392021    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:34.386097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:34.386045    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:34.386399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:34.386370    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:35.385938 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:35.385903    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:35.386311 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:35.385969    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:36.385527 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:36.385489    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:40:36.385686 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:36.385552    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:40:36.719805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:36.719696    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:36.919620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:36.919569    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:37.465679 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:37.465648    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:37.465974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:37.465952    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:37.466706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:40:37.466686    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:40:55.932911 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:40:55.932874    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:41:00.694347 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:00.694303    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:01.200916 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:01.200884    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:02.820972 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:02.820940    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:03.030519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:03.030483    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:14.423373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:14.423315    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:14.629766 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:14.629734    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:16.569160 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:16.569121    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:25.932817 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:41:25.932781    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:41:50.683759 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:50.683716    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:50.886604 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:50.886571    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:51.330103 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:51.330066    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:51.619404 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:51.619366    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:51.825613 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:51.825578    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:52.054541 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:52.054452    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:52.268580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:52.268534    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:52.528438 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:52.528390    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:52.861879 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:52.861827    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:53.212639 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:53.212544    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:53.417861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:53.417811    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:53.629999 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:53.629961    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:55.288146 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:55.288117    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:55.511321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:55.511275    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:55.934126 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:41:55.934090    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:41:56.354694 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:56.354666    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:56.560195 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:56.560163    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:57.066728 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:57.066698    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:57.276639 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:57.276607    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:58.655005 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:58.654970    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:58.860690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:58.860642    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:59.056999 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:59.056908    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:59.325408 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:59.325279    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:59.553095 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:59.553056    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:59.891390 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:41:59.891318    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:00.106824 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:00.106794    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:00.314699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:00.314624    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:00.653755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:00.653720    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:00.850502 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:00.850468    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:01.436555 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:01.436526    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:01.646392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:01.646360    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:01.908692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:01.908657    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:02.115530 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:02.115493    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:04.344126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:04.344092    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:04.540171 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:04.540141    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.533909 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:06.533875    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.742622 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:06.742574    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.945054 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:06.945023    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:09.348077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:09.348045    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:09.557398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:09.557368    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:09.805878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:09.805795    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:42:11.700940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:11.700903    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:11.916206 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:11.916170    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:12.138443 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:12.138404    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.451013 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:14.450983    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.643273 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:14.643242    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:16.003531 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.003452    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"]
Nov 03 20:42:16.007354 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.007318    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.053910 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.053879    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.054022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.053956    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.054096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.054076    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.054137 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.054125    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.056190 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.056174    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:42:16.102267 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.102246    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:42:16.102417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.102306    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1e6fe1-serving-cert\""
Nov 03 20:42:16.102417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.102370    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:42:16.102417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.102382    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\""
Nov 03 20:42:16.103947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.103931    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"]
Nov 03 20:42:16.154985 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.154960    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.155090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.155002    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.155090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.155057    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.155090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.155084    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.155249 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:16.155101    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-serving-cert: secret "maas-api-e2e-worker-w0-1e6fe1-serving-cert" not found
Nov 03 20:42:16.155249 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:16.155170    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls podName:409352e1-86cb-4b04-b253-6f6e8301576a nodeName:}" failed. No retries permitted until 2026-11-03 20:42:16.655154491 +0000 UTC m=+729.634711313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls") pod "maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" (UID: "409352e1-86cb-4b04-b253-6f6e8301576a") : secret "maas-api-e2e-worker-w0-1e6fe1-serving-cert" not found
Nov 03 20:42:16.155659 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.155638    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.157352 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.157318    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.214898 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.214870    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.249530 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.249505    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:16.449754 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.449725    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:16.658276 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.658240    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.660632 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.660605    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:16.892398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.892360    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:16.919269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:16.919243    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:17.109668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:17.109634    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:17.175621 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:17.175544    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:42:17.318487 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:17.318444    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:17.346836 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:17.346813    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"]
Nov 03 20:42:17.349079 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:42:17.349053    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409352e1_86cb_4b04_b253_6f6e8301576a.slice/crio-620929c1de3b9589b0b082c38e50bc3fead4fcc505cf7820cf3253e0231cbe89 WatchSource:0}: Error finding container 620929c1de3b9589b0b082c38e50bc3fead4fcc505cf7820cf3253e0231cbe89: Status 404 returned error can't find the container with id 620929c1de3b9589b0b082c38e50bc3fead4fcc505cf7820cf3253e0231cbe89
Nov 03 20:42:17.350546 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:17.350528    5318 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:42:17.584427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:17.584350    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" event={"ID":"409352e1-86cb-4b04-b253-6f6e8301576a","Type":"ContainerStarted","Data":"620929c1de3b9589b0b082c38e50bc3fead4fcc505cf7820cf3253e0231cbe89"}
Nov 03 20:42:18.591327 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:18.591291    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" event={"ID":"409352e1-86cb-4b04-b253-6f6e8301576a","Type":"ContainerStarted","Data":"ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388"}
Nov 03 20:42:18.591806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:18.591431    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:18.702140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:18.702090    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" podStartSLOduration=2.795152542 podStartE2EDuration="3.702076844s" totalImagesPullingTime="906.924302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:18.700742754 +0000 UTC m=+731.680299596" watchObservedRunningTime="2026-11-03 20:42:18.702076844 +0000 UTC m=+731.681633684"
Nov 03 20:42:20.871729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:20.871697    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:21.129279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:21.129204    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.144293 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.144259    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.358525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.358490    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.649183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.649152    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"]
Nov 03 20:42:22.653014 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.652990    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.705051 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.705025    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2a5ec9\""
Nov 03 20:42:22.709647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.709613    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7gq\" (UniqueName: \"kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq\") pod \"payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.709820 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.709711    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.748255 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.748228    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2a5ec9-dockercfg-kkczl\""
Nov 03 20:42:22.749749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.749730    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"]
Nov 03 20:42:22.810819 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.810794    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7gq\" (UniqueName: \"kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq\") pod \"payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.810965 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.810855    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.811570 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.811552    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.868194 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.868172    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7gq\" (UniqueName: \"kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq\") pod \"payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:22.966382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:22.966303    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:23.332403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:23.332377    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"]
Nov 03 20:42:23.334526 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:42:23.334501    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88ef88e_715e_4a3f_863f_25319fc0a555.slice/crio-35d568b9c9ee1eccb0fde30a136a724c9326376ffe389ebf6c83f2595d376845 WatchSource:0}: Error finding container 35d568b9c9ee1eccb0fde30a136a724c9326376ffe389ebf6c83f2595d376845: Status 404 returned error can't find the container with id 35d568b9c9ee1eccb0fde30a136a724c9326376ffe389ebf6c83f2595d376845
Nov 03 20:42:23.616094 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:23.616056    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" event={"ID":"a88ef88e-715e-4a3f-863f-25319fc0a555","Type":"ContainerStarted","Data":"35d568b9c9ee1eccb0fde30a136a724c9326376ffe389ebf6c83f2595d376845"}
Nov 03 20:42:24.602794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:24.602758    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 20:42:25.691606 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:25.691575    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:25.944277 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:25.944221    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:42:26.633846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:26.633807    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" event={"ID":"a88ef88e-715e-4a3f-863f-25319fc0a555","Type":"ContainerStarted","Data":"1afd293af81833d9e01e2400fa8ef272479cdac9afa556d5673232eccb70418a"}
Nov 03 20:42:26.633993 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:26.633935    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:26.744592 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:26.744532    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" podStartSLOduration=2.197544586 podStartE2EDuration="4.744515943s" totalImagesPullingTime="2.546971357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:26.744050383 +0000 UTC m=+739.723607226" watchObservedRunningTime="2026-11-03 20:42:26.744515943 +0000 UTC m=+739.724072783"
Nov 03 20:42:29.872494 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.872460    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"]
Nov 03 20:42:29.876067 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.876049    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:29.924555 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.924531    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-2a5ec9-serving-cert\""
Nov 03 20:42:29.972403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.972373    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"]
Nov 03 20:42:29.977160 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.977138    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:29.977226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.977184    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:29.977226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.977203    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdz7s\" (UniqueName: \"kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:29.977370 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:29.977354    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.078194 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.078161    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.078194 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.078195    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdz7s\" (UniqueName: \"kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.078471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.078252    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.078471 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.078285    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.078471 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:30.078412    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-serving-cert: secret "maas-api-e2e-worker-w5-2a5ec9-serving-cert" not found
Nov 03 20:42:30.078655 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:30.078481    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls podName:dd9837ff-b402-4a39-a327-0017091b0f43 nodeName:}" failed. No retries permitted until 2026-11-03 20:42:30.578457474 +0000 UTC m=+743.558014304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls") pod "maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" (UID: "dd9837ff-b402-4a39-a327-0017091b0f43") : secret "maas-api-e2e-worker-w5-2a5ec9-serving-cert" not found
Nov 03 20:42:30.079012 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.078989    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.080724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.080703    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.112224 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.112196    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:30.135781 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.135725    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdz7s\" (UniqueName: \"kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.305165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.305136    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:30.503235 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.503149    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:30.582595 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.582561    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.584891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.584857    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.789085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.788998    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:30.927472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:30.927440    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:31.156830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:31.156807    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"]
Nov 03 20:42:31.159153 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:42:31.159123    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9837ff_b402_4a39_a327_0017091b0f43.slice/crio-a5085b7d5461a48109dbcf7969eef8a518d3dfb3fe007b566b3183534d5ed61e WatchSource:0}: Error finding container a5085b7d5461a48109dbcf7969eef8a518d3dfb3fe007b566b3183534d5ed61e: Status 404 returned error can't find the container with id a5085b7d5461a48109dbcf7969eef8a518d3dfb3fe007b566b3183534d5ed61e
Nov 03 20:42:31.540399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:31.540302    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:31.657940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:31.657908    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" event={"ID":"dd9837ff-b402-4a39-a327-0017091b0f43","Type":"ContainerStarted","Data":"a5085b7d5461a48109dbcf7969eef8a518d3dfb3fe007b566b3183534d5ed61e"}
Nov 03 20:42:31.752905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:31.752880    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:32.664318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:32.664282    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" event={"ID":"dd9837ff-b402-4a39-a327-0017091b0f43","Type":"ContainerStarted","Data":"eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86"}
Nov 03 20:42:32.664733 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:32.664362    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:32.774869 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:32.774818    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" podStartSLOduration=2.920659667 podStartE2EDuration="3.774803547s" totalImagesPullingTime="854.14388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:32.774019268 +0000 UTC m=+745.753576134" watchObservedRunningTime="2026-11-03 20:42:32.774803547 +0000 UTC m=+745.754360386"
Nov 03 20:42:35.130593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.130533    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:42:35.204226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.204189    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:35.310304 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.310273    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"]
Nov 03 20:42:35.313947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.313926    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.366236 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.366207    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"openshift-service-ca.crt\""
Nov 03 20:42:35.366412 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.366252    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"kube-root-ca.crt\""
Nov 03 20:42:35.412218 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.412156    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"default-dockercfg-tv7ld\""
Nov 03 20:42:35.412417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.412392    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3b8feb\"/\"simulator-w2-3b8feb-kserve-self-signed-certs\""
Nov 03 20:42:35.415035 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.415014    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"]
Nov 03 20:42:35.424502 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.424475    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:35.428596 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.428574    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.428704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.428604    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.428704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.428641    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phvf\" (UniqueName: \"kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.428782 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.428733    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.428782 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.428769    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.428859 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.428790    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530235 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530200    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530299    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530328    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530409    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phvf\" (UniqueName: \"kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530448    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530694 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530496    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.530878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530847    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.531012 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530906    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.531012 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.530948    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.532982 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.532959    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.533259 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.533243    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.587810 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.587782    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phvf\" (UniqueName: \"kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf\") pod \"simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") " pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.626191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.626134    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:35.627592 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:35.627559    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:36.000512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:36.000488    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"]
Nov 03 20:42:36.003110 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:42:36.003082    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc35bbb_790d_42f9_9ffb_623c5761b34a.slice/crio-b5dc65fd0821922e84b12a8c5b553a2fa0eef1d426e643d59f94caeeb5b2c8db WatchSource:0}: Error finding container b5dc65fd0821922e84b12a8c5b553a2fa0eef1d426e643d59f94caeeb5b2c8db: Status 404 returned error can't find the container with id b5dc65fd0821922e84b12a8c5b553a2fa0eef1d426e643d59f94caeeb5b2c8db
Nov 03 20:42:36.688761 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:36.688719    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" event={"ID":"8fc35bbb-790d-42f9-9ffb-623c5761b34a","Type":"ContainerStarted","Data":"b5dc65fd0821922e84b12a8c5b553a2fa0eef1d426e643d59f94caeeb5b2c8db"}
Nov 03 20:42:37.641385 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:37.641354    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 20:42:38.677388 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:38.677354    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 20:42:38.699564 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:38.699527    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" event={"ID":"8fc35bbb-790d-42f9-9ffb-623c5761b34a","Type":"ContainerStarted","Data":"11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65"}
Nov 03 20:42:38.904796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:38.904748    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podStartSLOduration=2.162407193 podStartE2EDuration="3.904734212s" totalImagesPullingTime="1.742327019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:38.903524911 +0000 UTC m=+751.883081762" watchObservedRunningTime="2026-11-03 20:42:38.904734212 +0000 UTC m=+751.884291088"
Nov 03 20:42:39.626382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.626332    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:39.807993 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.807964    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"]
Nov 03 20:42:39.811535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.811511    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.860584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.860551    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-0cd49b-serving-cert\""
Nov 03 20:42:39.867923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.867890    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.868072 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.867946    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.868072 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.867972    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bpt\" (UniqueName: \"kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.868204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.868123    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.908488 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.908420    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"]
Nov 03 20:42:39.969182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.969150    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.969319 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.969211    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.969319 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.969235    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bpt\" (UniqueName: \"kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.969319 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.969290    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.969469 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:39.969375    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-serving-cert: secret "maas-api-e2e-worker-w1-0cd49b-serving-cert" not found
Nov 03 20:42:39.969469 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:39.969446    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls podName:d5db2eed-a5b9-4bc0-bc77-0f90911207cd nodeName:}" failed. No retries permitted until 2026-11-03 20:42:40.469428128 +0000 UTC m=+753.448984959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls") pod "maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" (UID: "d5db2eed-a5b9-4bc0-bc77-0f90911207cd") : secret "maas-api-e2e-worker-w1-0cd49b-serving-cert" not found
Nov 03 20:42:39.970026 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.970003    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:39.971518 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:39.971497    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:40.028974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:40.028946    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bpt\" (UniqueName: \"kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:40.059239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:40.059203    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:40.262455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:40.262384    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:40.474796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:40.474760    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:40.477080 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:40.477059    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:40.724669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:40.724636    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:41.090567 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:41.090540    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"]
Nov 03 20:42:41.093871 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:42:41.093845    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5db2eed_a5b9_4bc0_bc77_0f90911207cd.slice/crio-5032a825bb0e597e61a4e8ec0da97c32ff1026ca22b1c94333a69f8cb41134c9 WatchSource:0}: Error finding container 5032a825bb0e597e61a4e8ec0da97c32ff1026ca22b1c94333a69f8cb41134c9: Status 404 returned error can't find the container with id 5032a825bb0e597e61a4e8ec0da97c32ff1026ca22b1c94333a69f8cb41134c9
Nov 03 20:42:41.715094 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:41.715060    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" event={"ID":"d5db2eed-a5b9-4bc0-bc77-0f90911207cd","Type":"ContainerStarted","Data":"5032a825bb0e597e61a4e8ec0da97c32ff1026ca22b1c94333a69f8cb41134c9"}
Nov 03 20:42:42.722557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:42.722524    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" event={"ID":"d5db2eed-a5b9-4bc0-bc77-0f90911207cd","Type":"ContainerStarted","Data":"8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882"}
Nov 03 20:42:42.723038 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:42.722623    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:42.833571 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:42.833522    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" podStartSLOduration=2.983484206 podStartE2EDuration="3.833508538s" totalImagesPullingTime="850.024332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:42.831869625 +0000 UTC m=+755.811426464" watchObservedRunningTime="2026-11-03 20:42:42.833508538 +0000 UTC m=+755.813065377"
Nov 03 20:42:44.983252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:44.983209    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:45.626906 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:45.626861    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:45.640614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:45.640584    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:45.749587 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:45.749555    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:42:48.732787 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:48.732760    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 20:42:49.792450 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:49.792418    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:51.965511 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:51.965462    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:52.642426 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:52.642376    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:52.872079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:52.872022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:54.840719 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:54.840690    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:55.932898 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:42:55.932860    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:42:58.887179 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:58.887143    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:42:59.500578 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:42:59.500535    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:01.473530 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:01.473500    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:04.275655 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:04.275621    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:08.142801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:08.142775    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:08.720670 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:08.720631    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:08.938876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:08.938836    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:09.141755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:09.141718    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:11.206130 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:11.206100    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:11.775281 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:11.775247    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:11.971011 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:11.970974    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:13.870728 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:13.870687    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:16.833176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:16.833138    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:43:17.342112 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:17.342078    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:18.736836 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:18.736805    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:18.935545 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:18.935519    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:19.343183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:19.343157    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:19.553279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:19.553248    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:23.550561 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:23.550526    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:23.754015 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:23.753983    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:24.311553 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:24.311522    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:24.529890 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:24.529854    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:24.744891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:24.744854    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:25.932900 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:43:25.932837    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:43:28.389122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:28.389081    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:28.927302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:28.927265    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:29.518384 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:29.518344    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:29.723252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:29.723216    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:31.854731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:31.854698    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:43:33.069063 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:33.069032    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:35.492127 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:35.492086    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:37.580381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:37.580323    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:42.140470 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:42.140441    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:44.112096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.112064    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"]
Nov 03 20:43:44.115152 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.115137    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.164007 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.163982    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"kube-root-ca.crt\""
Nov 03 20:43:44.164007 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.163983    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"openshift-service-ca.crt\""
Nov 03 20:43:44.186593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.186559    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.186738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.186645    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrp8m\" (UniqueName: \"kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.186738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.186679    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.186855 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.186753    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.186855 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.186819    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.186855 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.186848    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.210122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.210098    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"default-dockercfg-7wjqd\""
Nov 03 20:43:44.210278 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.210252    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"simulator-w0-1e6fe1-kserve-self-signed-certs\""
Nov 03 20:43:44.211906 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.211887    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"]
Nov 03 20:43:44.281050 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.281022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:44.287259 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287239    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrp8m\" (UniqueName: \"kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287279    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287318    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287589 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287566    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287622    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287675    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287710    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.287958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.287939    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.288028 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.288008    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.289734 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.289713    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.289866 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.289847    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.348099 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.348067    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrp8m\" (UniqueName: \"kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m\") pod \"simulator-w0-1e6fe1-kserve-69cc484575-kskmb\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") " pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.445055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.444977    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:44.814865 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:44.814842    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"]
Nov 03 20:43:44.817054 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:43:44.817025    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod468fa626_3d72_4253_b637_4a6c06e954f4.slice/crio-13d5625d896c916d9d2f43c650524220850ecc7a98ac4e3fefe72af46b318872 WatchSource:0}: Error finding container 13d5625d896c916d9d2f43c650524220850ecc7a98ac4e3fefe72af46b318872: Status 404 returned error can't find the container with id 13d5625d896c916d9d2f43c650524220850ecc7a98ac4e3fefe72af46b318872
Nov 03 20:43:45.041533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:45.041501    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" event={"ID":"468fa626-3d72-4253-b637-4a6c06e954f4","Type":"ContainerStarted","Data":"dcb4c7e6326d46b461e3f494ca85aa4cd45d742136da970a8bb2416379e9640a"}
Nov 03 20:43:45.041702 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:45.041544    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" event={"ID":"468fa626-3d72-4253-b637-4a6c06e954f4","Type":"ContainerStarted","Data":"13d5625d896c916d9d2f43c650524220850ecc7a98ac4e3fefe72af46b318872"}
Nov 03 20:43:45.154272 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:45.154218    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" podStartSLOduration=1.154202387 podStartE2EDuration="1.154202387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:45.152016289 +0000 UTC m=+818.131573130" watchObservedRunningTime="2026-11-03 20:43:45.154202387 +0000 UTC m=+818.133759224"
Nov 03 20:43:45.445999 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:45.445919    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:46.589443 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:46.589399    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:51.243664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:51.243627    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:51.352863 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:51.352830    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:43:52.542596 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:52.542559    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:54.445651 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:54.445625    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:54.458122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:54.458102    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:55.107521 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:55.107489    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:43:55.678667 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:43:55.678630    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:55.933504 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:43:55.933430    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:44:00.175429 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:00.175398    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:05.070490 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:05.070453    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:05.280516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:05.280477    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:09.664368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:09.664317    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:10.692977 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:10.692947    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:44:11.817110 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.817077    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"]
Nov 03 20:44:11.820114 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.820095    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.890754 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.890722    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"openshift-service-ca.crt\""
Nov 03 20:44:11.915242 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.915214    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"default-dockercfg-8xb4k\""
Nov 03 20:44:11.915242 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.915227    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"kube-root-ca.crt\""
Nov 03 20:44:11.915457 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.915325    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"premium-w3-88228a-kserve-self-signed-certs\""
Nov 03 20:44:11.918307 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.918285    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"]
Nov 03 20:44:11.948226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.948203    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.948333 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.948244    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.948396 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.948350    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vff\" (UniqueName: \"kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.948444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.948400    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.948526 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.948507    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.948586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.948554    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:11.982405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:11.982332    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:12.049445 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049418    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049472    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049499    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049550    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049791 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049600    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049791 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049658    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vff\" (UniqueName: \"kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049959 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049928    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.049959 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.049943    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.050208 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.050186    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.051724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.051704    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.052133 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.052110    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.106756 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.106727    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vff\" (UniqueName: \"kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff\") pod \"premium-w3-88228a-kserve-7f7bc87958-dkq29\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") " pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.130650 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.130630    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:12.499745 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:12.499722    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"]
Nov 03 20:44:12.502073 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:44:12.502049    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf7441cb_533b_4e3c_90f5_1853bb3e090b.slice/crio-d4f9ec6591eebf575774890276acac434026e5d163844d6fb5cf6a4c6472f0c3 WatchSource:0}: Error finding container d4f9ec6591eebf575774890276acac434026e5d163844d6fb5cf6a4c6472f0c3: Status 404 returned error can't find the container with id d4f9ec6591eebf575774890276acac434026e5d163844d6fb5cf6a4c6472f0c3
Nov 03 20:44:13.196238 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:13.196200    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" event={"ID":"cf7441cb-533b-4e3c-90f5-1853bb3e090b","Type":"ContainerStarted","Data":"c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e"}
Nov 03 20:44:13.196772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:13.196251    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" event={"ID":"cf7441cb-533b-4e3c-90f5-1853bb3e090b","Type":"ContainerStarted","Data":"d4f9ec6591eebf575774890276acac434026e5d163844d6fb5cf6a4c6472f0c3"}
Nov 03 20:44:13.307374 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:13.307300    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podStartSLOduration=2.307286099 podStartE2EDuration="2.307286099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:13.305723828 +0000 UTC m=+846.285280681" watchObservedRunningTime="2026-11-03 20:44:13.307286099 +0000 UTC m=+846.286842938"
Nov 03 20:44:14.131568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:14.131525    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:14.162153 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:14.162121    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:17.234792 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:17.234766    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:19.538223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:19.538188    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:22.131113 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:22.131080    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:22.143144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:22.143125    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:22.250854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:22.250830    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:44:23.181627 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:23.181596    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:25.933139 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:44:25.933097    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:44:27.702243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:27.702209    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:32.168540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:32.168502    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:36.197187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:36.197150    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:44:36.866845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:36.866811    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:41.378174 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:41.378140    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:45.957663 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:45.957633    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:50.492447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:50.492409    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:51.920426 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:51.920393    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"]
Nov 03 20:44:51.923439 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:51.923419    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.018164 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018136    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"kube-root-ca.crt\""
Nov 03 20:44:52.018362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018296    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.018452 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018409    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"openshift-service-ca.crt\""
Nov 03 20:44:52.018452 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018443    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.018572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018489    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.018572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018503    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"default-dockercfg-8nct2\""
Nov 03 20:44:52.018572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018568    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"premium-w4-3d2008-kserve-self-signed-certs\""
Nov 03 20:44:52.018726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018582    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.018726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018623    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.018726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.018646    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.020422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.020398    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"]
Nov 03 20:44:52.087192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.087165    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:52.119403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119377    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.119520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119449    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.119520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119479    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.119653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119619    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.119710 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119649    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.119710 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119676    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.119931 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119906    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.120048 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.119960    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.120099 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.120057    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.121621 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.121604    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.121886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.121870    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.179706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.179644    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q\") pod \"premium-w4-3d2008-kserve-5469fcb47c-kkgcg\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") " pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.235620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.235594    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:52.610409 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:52.610387    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"]
Nov 03 20:44:52.613283 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:44:52.613254    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0577730_0066_4a22_9ab8_bb2b6d1fde11.slice/crio-faa5dabe1175f39244948fb17433b9a34148a3ba966bc8ece38f883d434f94a3 WatchSource:0}: Error finding container faa5dabe1175f39244948fb17433b9a34148a3ba966bc8ece38f883d434f94a3: Status 404 returned error can't find the container with id faa5dabe1175f39244948fb17433b9a34148a3ba966bc8ece38f883d434f94a3
Nov 03 20:44:53.397422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:53.397382    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" event={"ID":"c0577730-0066-4a22-9ab8-bb2b6d1fde11","Type":"ContainerStarted","Data":"31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4"}
Nov 03 20:44:53.397422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:53.397425    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" event={"ID":"c0577730-0066-4a22-9ab8-bb2b6d1fde11","Type":"ContainerStarted","Data":"faa5dabe1175f39244948fb17433b9a34148a3ba966bc8ece38f883d434f94a3"}
Nov 03 20:44:53.509759 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:53.509712    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podStartSLOduration=2.509697514 podStartE2EDuration="2.509697514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:53.50812286 +0000 UTC m=+886.487679700" watchObservedRunningTime="2026-11-03 20:44:53.509697514 +0000 UTC m=+886.489254354"
Nov 03 20:44:54.236096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:54.236060    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:44:55.120142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:55.120109    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:55.933553 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:44:55.933517    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:44:57.423726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:57.423692    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:44:59.861167 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:44:59.861140    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:00.192652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.192587    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"]
Nov 03 20:45:00.195318 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.195303    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.227815 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.227791    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"]
Nov 03 20:45:00.230417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.230402    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.245101 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.245077    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 20:45:00.299952 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.299926    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-b8stk\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.300078 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.300029    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcjt\" (UniqueName: \"kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt\") pod \"maas-api-key-cleanup-29895645-b8stk\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.300200 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.300141    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdvw\" (UniqueName: \"kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.300303 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.300201    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.331232 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.331161    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"]
Nov 03 20:45:00.336093 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.336049    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"]
Nov 03 20:45:00.401239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.401215    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdvw\" (UniqueName: \"kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.401427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.401278    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.401427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.401322    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-b8stk\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.401427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.401385    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcjt\" (UniqueName: \"kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt\") pod \"maas-api-key-cleanup-29895645-b8stk\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.402046 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.402024    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.402112 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.402098    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-b8stk\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.460077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.460011    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdvw\" (UniqueName: \"kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.460307 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.460284    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcjt\" (UniqueName: \"kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt\") pod \"maas-api-key-cleanup-29895645-b8stk\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.507478 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.507446    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:00.540947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.540917    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:00.884608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.884535    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"]
Nov 03 20:45:00.887382 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:45:00.887350    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820c79d8_9de0_4110_a2c5_0704bd986885.slice/crio-ad5ccd4871d56a86e6d4224e9675b2805b9bd4de766bcda8b4235f1341bd2d38 WatchSource:0}: Error finding container ad5ccd4871d56a86e6d4224e9675b2805b9bd4de766bcda8b4235f1341bd2d38: Status 404 returned error can't find the container with id ad5ccd4871d56a86e6d4224e9675b2805b9bd4de766bcda8b4235f1341bd2d38
Nov 03 20:45:00.936225 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:00.936203    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"]
Nov 03 20:45:00.937635 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:45:00.937612    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436e5567_5cce_438e_9d1e_597edb9169d4.slice/crio-9372333c8be0cbc8c8b5c13cda2527c2807b1bfe7afc273ef8f39ca02ad564ed WatchSource:0}: Error finding container 9372333c8be0cbc8c8b5c13cda2527c2807b1bfe7afc273ef8f39ca02ad564ed: Status 404 returned error can't find the container with id 9372333c8be0cbc8c8b5c13cda2527c2807b1bfe7afc273ef8f39ca02ad564ed
Nov 03 20:45:01.441088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:01.441042    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" event={"ID":"436e5567-5cce-438e-9d1e-597edb9169d4","Type":"ContainerStarted","Data":"9372333c8be0cbc8c8b5c13cda2527c2807b1bfe7afc273ef8f39ca02ad564ed"}
Nov 03 20:45:01.442321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:01.442290    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" event={"ID":"820c79d8-9de0-4110-a2c5-0704bd986885","Type":"ContainerStarted","Data":"ad5ccd4871d56a86e6d4224e9675b2805b9bd4de766bcda8b4235f1341bd2d38"}
Nov 03 20:45:02.235963 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:02.235930    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:45:02.248955 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:02.248933    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:45:02.459003 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:02.458976    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:45:04.565416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:04.565384    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:06.473110 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:06.473074    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" event={"ID":"820c79d8-9de0-4110-a2c5-0704bd986885","Type":"ContainerStarted","Data":"3b7e63db43e063035245b7be6153493bd36ac629bcfdae73881a1809b9211aa0"}
Nov 03 20:45:06.474749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:06.474721    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" event={"ID":"436e5567-5cce-438e-9d1e-597edb9169d4","Type":"ContainerStarted","Data":"1dcff52dbb97ec2c3a9d5a75a471b9b6bfa30df00597a1efd83f1f52a3d4e511"}
Nov 03 20:45:06.583123 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:06.583079    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" podStartSLOduration=1.618260144 podStartE2EDuration="6.583066749s" totalImagesPullingTime="4.964806605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:06.580679057 +0000 UTC m=+899.560235897" watchObservedRunningTime="2026-11-03 20:45:06.583066749 +0000 UTC m=+899.562623589"
Nov 03 20:45:06.688618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:06.688576    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" podStartSLOduration=1.9088526959999998 podStartE2EDuration="6.688562981s" totalImagesPullingTime="4.779710285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:06.687497415 +0000 UTC m=+899.667054254" watchObservedRunningTime="2026-11-03 20:45:06.688562981 +0000 UTC m=+899.668119821"
Nov 03 20:45:07.410467 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.410438    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:45:07.414935 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.414913    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:45:07.481211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.481188    5318 generic.go:356] "Generic (PLEG): container finished" podID="436e5567-5cce-438e-9d1e-597edb9169d4" containerID="1dcff52dbb97ec2c3a9d5a75a471b9b6bfa30df00597a1efd83f1f52a3d4e511" exitCode=60
Nov 03 20:45:07.481565 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.481281    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" event={"ID":"436e5567-5cce-438e-9d1e-597edb9169d4","Type":"ContainerDied","Data":"1dcff52dbb97ec2c3a9d5a75a471b9b6bfa30df00597a1efd83f1f52a3d4e511"}
Nov 03 20:45:07.481565 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.481498    5318 scope.go:122] "RemoveContainer" containerID="1dcff52dbb97ec2c3a9d5a75a471b9b6bfa30df00597a1efd83f1f52a3d4e511"
Nov 03 20:45:07.483027 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.483001    5318 generic.go:356] "Generic (PLEG): container finished" podID="820c79d8-9de0-4110-a2c5-0704bd986885" containerID="3b7e63db43e063035245b7be6153493bd36ac629bcfdae73881a1809b9211aa0" exitCode=60
Nov 03 20:45:07.483112 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.483029    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" event={"ID":"820c79d8-9de0-4110-a2c5-0704bd986885","Type":"ContainerDied","Data":"3b7e63db43e063035245b7be6153493bd36ac629bcfdae73881a1809b9211aa0"}
Nov 03 20:45:07.483230 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:07.483217    5318 scope.go:122] "RemoveContainer" containerID="3b7e63db43e063035245b7be6153493bd36ac629bcfdae73881a1809b9211aa0"
Nov 03 20:45:08.490273 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.490206    5318 generic.go:356] "Generic (PLEG): container finished" podID="820c79d8-9de0-4110-a2c5-0704bd986885" containerID="6dcd44c6b14eec109dba1d0bfea26a49aa95b9260f74af07ec52d483ed681e05" exitCode=60
Nov 03 20:45:08.490685 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.490284    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" event={"ID":"820c79d8-9de0-4110-a2c5-0704bd986885","Type":"ContainerDied","Data":"6dcd44c6b14eec109dba1d0bfea26a49aa95b9260f74af07ec52d483ed681e05"}
Nov 03 20:45:08.490685 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.490365    5318 scope.go:122] "RemoveContainer" containerID="3b7e63db43e063035245b7be6153493bd36ac629bcfdae73881a1809b9211aa0"
Nov 03 20:45:08.490685 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.490497    5318 scope.go:122] "RemoveContainer" containerID="6dcd44c6b14eec109dba1d0bfea26a49aa95b9260f74af07ec52d483ed681e05"
Nov 03 20:45:08.490852 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:08.490737    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-b8stk_odh-ai-gateway-infra(820c79d8-9de0-4110-a2c5-0704bd986885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" podUID="820c79d8-9de0-4110-a2c5-0704bd986885"
Nov 03 20:45:08.492266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.492246    5318 generic.go:356] "Generic (PLEG): container finished" podID="436e5567-5cce-438e-9d1e-597edb9169d4" containerID="6dd0c67867e7634ac53327372a20fae00a58e1ab8504914595c5b66036b1d2ab" exitCode=60
Nov 03 20:45:08.492378 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.492271    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" event={"ID":"436e5567-5cce-438e-9d1e-597edb9169d4","Type":"ContainerDied","Data":"6dd0c67867e7634ac53327372a20fae00a58e1ab8504914595c5b66036b1d2ab"}
Nov 03 20:45:08.492527 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.492514    5318 scope.go:122] "RemoveContainer" containerID="6dd0c67867e7634ac53327372a20fae00a58e1ab8504914595c5b66036b1d2ab"
Nov 03 20:45:08.492708 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:08.492689    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp_odh-ai-gateway-infra(436e5567-5cce-438e-9d1e-597edb9169d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" podUID="436e5567-5cce-438e-9d1e-597edb9169d4"
Nov 03 20:45:08.503753 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:08.503733    5318 scope.go:122] "RemoveContainer" containerID="1dcff52dbb97ec2c3a9d5a75a471b9b6bfa30df00597a1efd83f1f52a3d4e511"
Nov 03 20:45:09.288774 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:09.288738    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:09.499581 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:09.499549    5318 scope.go:122] "RemoveContainer" containerID="6dd0c67867e7634ac53327372a20fae00a58e1ab8504914595c5b66036b1d2ab"
Nov 03 20:45:09.499999 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:09.499787    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp_odh-ai-gateway-infra(436e5567-5cce-438e-9d1e-597edb9169d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" podUID="436e5567-5cce-438e-9d1e-597edb9169d4"
Nov 03 20:45:09.501371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:09.501349    5318 scope.go:122] "RemoveContainer" containerID="6dcd44c6b14eec109dba1d0bfea26a49aa95b9260f74af07ec52d483ed681e05"
Nov 03 20:45:09.501649 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:09.501627    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-b8stk_odh-ai-gateway-infra(820c79d8-9de0-4110-a2c5-0704bd986885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" podUID="820c79d8-9de0-4110-a2c5-0704bd986885"
Nov 03 20:45:13.795559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:13.795529    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:15.407991 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:15.407955    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:17.947599 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:17.947565    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:18.224607 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.224529    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"]
Nov 03 20:45:18.228364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.228325    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.267376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.267326    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.267481 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.267379    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.267522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.267497    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsjc\" (UniqueName: \"kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.302262 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.302232    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:18.304275 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.304250    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"]
Nov 03 20:45:18.368694 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.368666    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.368830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.368700    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.368830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.368731    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsjc\" (UniqueName: \"kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.369458 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.369435    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.371026 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.371005    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.397848 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.397811    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:18.436257 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.436222    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsjc\" (UniqueName: \"kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc\") pod \"maas-api-revoke-sub-39d801c4-wcn6n\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.539762 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.539687    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:18.907322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:18.907297    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"]
Nov 03 20:45:18.910095 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:45:18.910068    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdca8312_e27b_409d_a83b_58895ef422de.slice/crio-e9f4e33f6ea0b2a28f24bb12ad4d11ac1d0e49b96265afa180c10bf53d8cc46c WatchSource:0}: Error finding container e9f4e33f6ea0b2a28f24bb12ad4d11ac1d0e49b96265afa180c10bf53d8cc46c: Status 404 returned error can't find the container with id e9f4e33f6ea0b2a28f24bb12ad4d11ac1d0e49b96265afa180c10bf53d8cc46c
Nov 03 20:45:19.554958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:19.554870    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n" event={"ID":"cdca8312-e27b-409d-a83b-58895ef422de","Type":"ContainerStarted","Data":"22630703e39bac393239caa2846ef6d5047fe1dad5fef68eadbf08134d7b84c0"}
Nov 03 20:45:19.554958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:19.554925    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n" event={"ID":"cdca8312-e27b-409d-a83b-58895ef422de","Type":"ContainerStarted","Data":"e9f4e33f6ea0b2a28f24bb12ad4d11ac1d0e49b96265afa180c10bf53d8cc46c"}
Nov 03 20:45:19.664238 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:19.664180    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n" podStartSLOduration=1.6641617389999999 podStartE2EDuration="1.664161739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:19.662946468 +0000 UTC m=+912.642503308" watchObservedRunningTime="2026-11-03 20:45:19.664161739 +0000 UTC m=+912.643718581"
Nov 03 20:45:20.248183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:20.248152    5318 scope.go:122] "RemoveContainer" containerID="6dd0c67867e7634ac53327372a20fae00a58e1ab8504914595c5b66036b1d2ab"
Nov 03 20:45:20.248418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:20.248397    5318 scope.go:122] "RemoveContainer" containerID="6dcd44c6b14eec109dba1d0bfea26a49aa95b9260f74af07ec52d483ed681e05"
Nov 03 20:45:20.561644 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:20.561573    5318 generic.go:356] "Generic (PLEG): container finished" podID="cdca8312-e27b-409d-a83b-58895ef422de" containerID="22630703e39bac393239caa2846ef6d5047fe1dad5fef68eadbf08134d7b84c0" exitCode=0
Nov 03 20:45:20.562022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:20.561649    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n" event={"ID":"cdca8312-e27b-409d-a83b-58895ef422de","Type":"ContainerDied","Data":"22630703e39bac393239caa2846ef6d5047fe1dad5fef68eadbf08134d7b84c0"}
Nov 03 20:45:21.569178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.569141    5318 generic.go:356] "Generic (PLEG): container finished" podID="820c79d8-9de0-4110-a2c5-0704bd986885" containerID="35ec095f1adf286a353882ac9e04d76b6a7f6a95766124e150e023b665c989d1" exitCode=60
Nov 03 20:45:21.569671 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.569207    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" event={"ID":"820c79d8-9de0-4110-a2c5-0704bd986885","Type":"ContainerDied","Data":"35ec095f1adf286a353882ac9e04d76b6a7f6a95766124e150e023b665c989d1"}
Nov 03 20:45:21.569671 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.569269    5318 scope.go:122] "RemoveContainer" containerID="6dcd44c6b14eec109dba1d0bfea26a49aa95b9260f74af07ec52d483ed681e05"
Nov 03 20:45:21.569671 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.569638    5318 scope.go:122] "RemoveContainer" containerID="35ec095f1adf286a353882ac9e04d76b6a7f6a95766124e150e023b665c989d1"
Nov 03 20:45:21.569934 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:21.569907    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-b8stk_odh-ai-gateway-infra(820c79d8-9de0-4110-a2c5-0704bd986885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" podUID="820c79d8-9de0-4110-a2c5-0704bd986885"
Nov 03 20:45:21.573055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.573029    5318 generic.go:356] "Generic (PLEG): container finished" podID="436e5567-5cce-438e-9d1e-597edb9169d4" containerID="116b43da27b32dd40159ea72228d83f2a39682fd30945b29c9e7470791e3fd1d" exitCode=60
Nov 03 20:45:21.573181 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.573107    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" event={"ID":"436e5567-5cce-438e-9d1e-597edb9169d4","Type":"ContainerDied","Data":"116b43da27b32dd40159ea72228d83f2a39682fd30945b29c9e7470791e3fd1d"}
Nov 03 20:45:21.573613 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.573597    5318 scope.go:122] "RemoveContainer" containerID="116b43da27b32dd40159ea72228d83f2a39682fd30945b29c9e7470791e3fd1d"
Nov 03 20:45:21.573869 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:21.573843    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp_odh-ai-gateway-infra(436e5567-5cce-438e-9d1e-597edb9169d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" podUID="436e5567-5cce-438e-9d1e-597edb9169d4"
Nov 03 20:45:21.581511 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.581490    5318 scope.go:122] "RemoveContainer" containerID="6dd0c67867e7634ac53327372a20fae00a58e1ab8504914595c5b66036b1d2ab"
Nov 03 20:45:21.729232 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.729210    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:21.802592 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.802559    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca\") pod \"cdca8312-e27b-409d-a83b-58895ef422de\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") "
Nov 03 20:45:21.802745 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.802600    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc\" (UniqueName: \"kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc\") pod \"cdca8312-e27b-409d-a83b-58895ef422de\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") "
Nov 03 20:45:21.802807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.802740    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca\") pod \"cdca8312-e27b-409d-a83b-58895ef422de\" (UID: \"cdca8312-e27b-409d-a83b-58895ef422de\") "
Nov 03 20:45:21.802961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.802941    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca" pod "cdca8312-e27b-409d-a83b-58895ef422de" (UID: "cdca8312-e27b-409d-a83b-58895ef422de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:21.803117 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.803099    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:21.804858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.804833    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca" pod "cdca8312-e27b-409d-a83b-58895ef422de" (UID: "cdca8312-e27b-409d-a83b-58895ef422de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:21.805070 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.805055    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc" pod "cdca8312-e27b-409d-a83b-58895ef422de" (UID: "cdca8312-e27b-409d-a83b-58895ef422de"). InnerVolumeSpecName "kube-api-access-vbsjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:21.904689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.904656    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdca8312-e27b-409d-a83b-58895ef422de-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:21.904689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:21.904690    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbsjc\" (UniqueName: \"kubernetes.io/projected/cdca8312-e27b-409d-a83b-58895ef422de-kube-api-access-vbsjc\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:22.582821 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.582784    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n" event={"ID":"cdca8312-e27b-409d-a83b-58895ef422de","Type":"ContainerDied","Data":"e9f4e33f6ea0b2a28f24bb12ad4d11ac1d0e49b96265afa180c10bf53d8cc46c"}
Nov 03 20:45:22.583246 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.582847    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9f4e33f6ea0b2a28f24bb12ad4d11ac1d0e49b96265afa180c10bf53d8cc46c"
Nov 03 20:45:22.583246 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.582796    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"
Nov 03 20:45:22.693970 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.693922    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"]
Nov 03 20:45:22.847618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.847596    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:22.850519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.850501    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"]
Nov 03 20:45:22.916698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.916664    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt\" (UniqueName: \"kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt\") pod \"820c79d8-9de0-4110-a2c5-0704bd986885\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") "
Nov 03 20:45:22.916840 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.916786    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca\") pod \"820c79d8-9de0-4110-a2c5-0704bd986885\" (UID: \"820c79d8-9de0-4110-a2c5-0704bd986885\") "
Nov 03 20:45:22.917139 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.917114    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca" pod "820c79d8-9de0-4110-a2c5-0704bd986885" (UID: "820c79d8-9de0-4110-a2c5-0704bd986885"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:22.919920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.919894    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt" pod "820c79d8-9de0-4110-a2c5-0704bd986885" (UID: "820c79d8-9de0-4110-a2c5-0704bd986885"). InnerVolumeSpecName "kube-api-access-xpcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:22.997877 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.997848    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:22.998852 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:22.998832    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:23.017731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.017705    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcjt\" (UniqueName: \"kubernetes.io/projected/820c79d8-9de0-4110-a2c5-0704bd986885-kube-api-access-xpcjt\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:23.017731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.017728    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/820c79d8-9de0-4110-a2c5-0704bd986885-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:23.118559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.118472    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw\" (UniqueName: \"kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw\") pod \"436e5567-5cce-438e-9d1e-597edb9169d4\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") "
Nov 03 20:45:23.118559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.118539    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca\") pod \"436e5567-5cce-438e-9d1e-597edb9169d4\" (UID: \"436e5567-5cce-438e-9d1e-597edb9169d4\") "
Nov 03 20:45:23.118928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.118901    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca" pod "436e5567-5cce-438e-9d1e-597edb9169d4" (UID: "436e5567-5cce-438e-9d1e-597edb9169d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:23.120701 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.120674    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw" pod "436e5567-5cce-438e-9d1e-597edb9169d4" (UID: "436e5567-5cce-438e-9d1e-597edb9169d4"). InnerVolumeSpecName "kube-api-access-tmdvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:23.159675 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.159647    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"]
Nov 03 20:45:23.160050 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160038    5318 cpu_manager.go:414] "containerMap: removing container" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:45:23.160097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160051    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:45:23.160097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160072    5318 cpu_manager.go:414] "containerMap: removing container" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:45:23.160097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160078    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:45:23.160097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160089    5318 cpu_manager.go:414] "containerMap: removing container" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160094    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160107    5318 cpu_manager.go:414] "containerMap: removing container" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160112    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160119    5318 cpu_manager.go:414] "containerMap: removing container" podUID="cdca8312-e27b-409d-a83b-58895ef422de" containerName="revoke-keys"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160123    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdca8312-e27b-409d-a83b-58895ef422de" containerName="revoke-keys"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160184    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160192    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160199    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160205    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdca8312-e27b-409d-a83b-58895ef422de" containerName="revoke-keys"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160211    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:45:23.160247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.160219    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:45:23.162465 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.162451    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.212099 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.212064    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"kube-root-ca.crt\""
Nov 03 20:45:23.212269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.212130    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"openshift-service-ca.crt\""
Nov 03 20:45:23.212445 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.212427    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"default-dockercfg-9dx65\""
Nov 03 20:45:23.212565 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.212545    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"premium-w5-2a5ec9-kserve-self-signed-certs\""
Nov 03 20:45:23.218872 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.218828    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"]
Nov 03 20:45:23.219394 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219304    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.219510 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219439    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.219568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219525    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.219609 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219570    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.219664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219617    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dt4\" (UniqueName: \"kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.219698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219674    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.219758 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219744    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdvw\" (UniqueName: \"kubernetes.io/projected/436e5567-5cce-438e-9d1e-597edb9169d4-kube-api-access-tmdvw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:23.219831 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.219803    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436e5567-5cce-438e-9d1e-597edb9169d4-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:23.320602 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.320540    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321011 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.320633    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321011 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.320694    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321011 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.320927    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.321009    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.321083    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.321086    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dt4\" (UniqueName: \"kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.321234    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.321885 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.321861    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.322615 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.322588    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:23.323765 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.323744    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.324158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.324139    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.383650 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.383572    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dt4\" (UniqueName: \"kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4\") pod \"premium-w5-2a5ec9-kserve-7bc7965bd-plhz7\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") " pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.476295 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.476259    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:23.602216 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.601816    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"
Nov 03 20:45:23.602216 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.601828    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk" event={"ID":"820c79d8-9de0-4110-a2c5-0704bd986885","Type":"ContainerDied","Data":"ad5ccd4871d56a86e6d4224e9675b2805b9bd4de766bcda8b4235f1341bd2d38"}
Nov 03 20:45:23.602216 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.601901    5318 scope.go:122] "RemoveContainer" containerID="35ec095f1adf286a353882ac9e04d76b6a7f6a95766124e150e023b665c989d1"
Nov 03 20:45:23.604072 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.603871    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp" event={"ID":"436e5567-5cce-438e-9d1e-597edb9169d4","Type":"ContainerDied","Data":"9372333c8be0cbc8c8b5c13cda2527c2807b1bfe7afc273ef8f39ca02ad564ed"}
Nov 03 20:45:23.604072 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.603933    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"
Nov 03 20:45:23.614222 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.614206    5318 scope.go:122] "RemoveContainer" containerID="116b43da27b32dd40159ea72228d83f2a39682fd30945b29c9e7470791e3fd1d"
Nov 03 20:45:23.755344 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.755271    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"]
Nov 03 20:45:23.761193 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.761169    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-b8stk"]
Nov 03 20:45:23.869778 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.869754    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"]
Nov 03 20:45:23.872168 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.872143    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"]
Nov 03 20:45:23.872398 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:45:23.872376    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4526ab7_af1a_43eb_be89_429e62e1c5f9.slice/crio-40add25b7455fe96ae13048d4988b0ea35ef6daaad4fe6a9d29367b97ec0fcea WatchSource:0}: Error finding container 40add25b7455fe96ae13048d4988b0ea35ef6daaad4fe6a9d29367b97ec0fcea: Status 404 returned error can't find the container with id 40add25b7455fe96ae13048d4988b0ea35ef6daaad4fe6a9d29367b97ec0fcea
Nov 03 20:45:23.880479 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:23.880119    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895645-z48kp"]
Nov 03 20:45:24.612525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:24.612495    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" event={"ID":"b4526ab7-af1a-43eb-be89-429e62e1c5f9","Type":"ContainerStarted","Data":"574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a"}
Nov 03 20:45:24.612895 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:24.612534    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" event={"ID":"b4526ab7-af1a-43eb-be89-429e62e1c5f9","Type":"ContainerStarted","Data":"40add25b7455fe96ae13048d4988b0ea35ef6daaad4fe6a9d29367b97ec0fcea"}
Nov 03 20:45:24.722264 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:24.722215    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podStartSLOduration=1.722199354 podStartE2EDuration="1.722199354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:24.722036686 +0000 UTC m=+917.701593527" watchObservedRunningTime="2026-11-03 20:45:24.722199354 +0000 UTC m=+917.701756194"
Nov 03 20:45:25.254604 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:25.254574    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" path="/var/lib/kubelet/pods/436e5567-5cce-438e-9d1e-597edb9169d4/volumes"
Nov 03 20:45:25.255017 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:25.255001    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" path="/var/lib/kubelet/pods/820c79d8-9de0-4110-a2c5-0704bd986885/volumes"
Nov 03 20:45:25.477247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:25.477218    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:25.939294 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:25.939259    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:45:27.581587 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:27.581552    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:30.823600 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:30.823565    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:32.069492 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:32.069460    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:33.476979 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:33.476947    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:33.490411 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:33.490387    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:33.670324 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:33.670280    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:45:36.163201 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:36.163169    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:45:36.581931 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:36.581854    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:38.460708 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:38.460680    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:38.664982 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:38.664928    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:40.324406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:40.324369    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:40.526445 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:40.526410    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:41.086648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:41.086616    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:43.748331 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:43.748299    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.588444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:45.588416    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:50.200489 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:50.200445    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:54.675992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:54.675951    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:55.936368 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:45:55.936310    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:45:57.460779 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:57.460750    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"]
Nov 03 20:45:57.461210 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:57.460932    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy" containerID="cri-o://9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f" gracePeriod=30
Nov 03 20:45:59.296769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:45:59.296737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:00.501002 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:00.500965    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:00.698655 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:00.698619    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:02.712493 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.712471    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:46:02.790691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790617    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.790691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790681    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.790908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790717    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.790908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790755    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.790908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790782    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.790908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790810    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.790908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790835    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.791269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790887    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.791357 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791310    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy\") pod \"afe170fd-b1c2-413a-b80d-59e406f5e403\" (UID: \"afe170fd-b1c2-413a-b80d-59e406f5e403\") "
Nov 03 20:46:02.791403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.790935    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:02.791403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791051    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:02.791403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791142    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:02.791403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791165    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:02.791605 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791186    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:02.791661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791632    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.791661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791646    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.791661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791655    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.791765 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791667    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe170fd-b1c2-413a-b80d-59e406f5e403-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.791765 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.791683    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.793147 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.793116    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:02.793269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.793247    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:02.793430 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.793317    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:46:02.793430 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.793348    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h" pod "afe170fd-b1c2-413a-b80d-59e406f5e403" (UID: "afe170fd-b1c2-413a-b80d-59e406f5e403"). InnerVolumeSpecName "kube-api-access-rpk5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:02.818283 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.818261    5318 generic.go:356] "Generic (PLEG): container finished" podID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerID="9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f" exitCode=0
Nov 03 20:46:02.818400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.818305    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"
Nov 03 20:46:02.818400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.818363    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" event={"ID":"afe170fd-b1c2-413a-b80d-59e406f5e403","Type":"ContainerDied","Data":"9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f"}
Nov 03 20:46:02.818477 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.818416    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg" event={"ID":"afe170fd-b1c2-413a-b80d-59e406f5e403","Type":"ContainerDied","Data":"18c85c3860a33c07863fe0dcadac2d15e43e93c2ed031a331f65b4f51098bd4b"}
Nov 03 20:46:02.818477 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.818441    5318 scope.go:122] "RemoveContainer" containerID="9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f"
Nov 03 20:46:02.827783 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.827769    5318 scope.go:122] "RemoveContainer" containerID="9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f"
Nov 03 20:46:02.828026 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:46:02.828009    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f\": container with ID starting with 9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f not found: ID does not exist" containerID="9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f"
Nov 03 20:46:02.828085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.828037    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f"} err="failed to get container status \"9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f\": rpc error: code = NotFound desc = could not find container \"9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f\": container with ID starting with 9a161a5de8d3f36cdc34052c7c2b9d003ddd97be8e1b075ecd8f5d09857d336f not found: ID does not exist"
Nov 03 20:46:02.892948 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.892926    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpk5h\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-kube-api-access-rpk5h\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.893056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.892950    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.893056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.892964    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.893056 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.892975    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe170fd-b1c2-413a-b80d-59e406f5e403-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:02.985096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.985061    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"]
Nov 03 20:46:02.988088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:02.988063    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dfdmg"]
Nov 03 20:46:03.259080 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:03.259038    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" path="/var/lib/kubelet/pods/afe170fd-b1c2-413a-b80d-59e406f5e403/volumes"
Nov 03 20:46:03.932524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:03.932494    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:08.481833 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.481790    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:08.489211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489184    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"]
Nov 03 20:46:08.489681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489664    5318 cpu_manager.go:414] "containerMap: removing container" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy"
Nov 03 20:46:08.489788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489683    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy"
Nov 03 20:46:08.489788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489717    5318 cpu_manager.go:414] "containerMap: removing container" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:46:08.489788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489726    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:46:08.489788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489755    5318 cpu_manager.go:414] "containerMap: removing container" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:46:08.489788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489762    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820c79d8-9de0-4110-a2c5-0704bd986885" containerName="cleanup"
Nov 03 20:46:08.490038 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489878    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe170fd-b1c2-413a-b80d-59e406f5e403" containerName="istio-proxy"
Nov 03 20:46:08.490038 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.489894    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="436e5567-5cce-438e-9d1e-597edb9169d4" containerName="cleanup"
Nov 03 20:46:08.493682 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.493662    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.542680 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.542659    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-88228a-openshift-default-dockercfg-tb65c\""
Nov 03 20:46:08.550096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550067    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550102    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550128    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550306 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550288    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550424 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550404    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550474 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550445    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550646 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550612    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwpg\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.550646 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.550683    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.551120 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.551090    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.588623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.588599    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:46:08.590992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.590966    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"]
Nov 03 20:46:08.651949 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.651923    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.651957    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.651980    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652004    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652041    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652061    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652087    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwpg\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652129    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652435 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652390    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652435 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652421    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652536 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652452    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652694    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652707    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.652841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.652811    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.654443 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.654423    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.654691 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.654669    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.709036 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.709012    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.709421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.709398    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwpg\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:08.806314 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:08.806236    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:09.223057 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:09.223028    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"]
Nov 03 20:46:09.225183 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:46:09.225155    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ee868d_2a9f_4ffb_84e9_ff2afa6b6142.slice/crio-dd9333da6ee30300accecd96690421a95ba2db9c53f6a1979a24f1d591bcf806 WatchSource:0}: Error finding container dd9333da6ee30300accecd96690421a95ba2db9c53f6a1979a24f1d591bcf806: Status 404 returned error can't find the container with id dd9333da6ee30300accecd96690421a95ba2db9c53f6a1979a24f1d591bcf806
Nov 03 20:46:09.227608 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:09.227577    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:46:09.227696 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:09.227637    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:46:09.227696 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:09.227666    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:46:09.859439 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:09.859400    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" event={"ID":"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142","Type":"ContainerStarted","Data":"7afbcf1629deb1d439c57d8dc4c402f4c77bb23ffae4efe4d9efde374bfaa2d8"}
Nov 03 20:46:09.859816 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:09.859459    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" event={"ID":"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142","Type":"ContainerStarted","Data":"dd9333da6ee30300accecd96690421a95ba2db9c53f6a1979a24f1d591bcf806"}
Nov 03 20:46:10.018899 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:10.018847    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podStartSLOduration=2.018827282 podStartE2EDuration="2.018827282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:10.017801666 +0000 UTC m=+962.997358506" watchObservedRunningTime="2026-11-03 20:46:10.018827282 +0000 UTC m=+962.998384123"
Nov 03 20:46:10.806607 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:10.806572    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:10.808030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:10.808003    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:46:10.808151 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:10.808061    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:46:11.806985 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:11.806950    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:46:11.807470 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:11.807023    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:46:12.447539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:12.447509    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:46:12.806912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:12.806819    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:46:12.806912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:12.806884    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:46:13.303704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:13.303668    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:13.806928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:13.806894    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:46:13.807094 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:13.806959    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:46:14.807288 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:14.807251    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:46:14.807782 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:14.807311    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:46:16.669719 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:16.669688    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:16.670840 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:16.670810    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:16.671722 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:16.671701    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 20:46:18.068605 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:18.068565    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:18.303223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:18.303188    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:22.681035 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:22.681004    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:25.125125 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:25.125082    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:25.932955 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:46:25.932921    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:46:27.239382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:27.239330    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:31.748718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:31.748682    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:36.269629 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:36.269589    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:40.778827 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:40.778793    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:45.460411 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:45.460329    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:50.062242 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:50.062203    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:54.586944 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:54.586912    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:55.934228 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:46:55.934191    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:46:59.081127 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.081095    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:59.547945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.547910    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"]
Nov 03 20:46:59.551504 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.551480    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.567147 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.567120    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.567263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.567166    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.567314 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.567262    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp4cs\" (UniqueName: \"kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.600698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.600668    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:59.600904 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.600784    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 20:46:59.651023 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.650983    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"]
Nov 03 20:46:59.668195 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.668157    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.668367 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.668209    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.668367 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.668271    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp4cs\" (UniqueName: \"kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.668933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.668907    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.670454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.670426    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.737406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.737372    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp4cs\" (UniqueName: \"kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs\") pod \"maas-api-revoke-sub-e8d05cb2-wqk66\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:46:59.866789 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:46:59.866761    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:47:00.242614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.242589    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"]
Nov 03 20:47:00.245881 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:47:00.245855    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4721126a_d449_4123_9ac8_e8e215cd68f3.slice/crio-50486318b7c9cfe44f87ae1b1e3525981879a876c2a507c7e61dd97e24ad0e0f WatchSource:0}: Error finding container 50486318b7c9cfe44f87ae1b1e3525981879a876c2a507c7e61dd97e24ad0e0f: Status 404 returned error can't find the container with id 50486318b7c9cfe44f87ae1b1e3525981879a876c2a507c7e61dd97e24ad0e0f
Nov 03 20:47:00.502881 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.502814    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"]
Nov 03 20:47:00.507680 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.507660    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.563997 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.563963    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"]
Nov 03 20:47:00.574601 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.574577    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.574714 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.574622    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5c8\" (UniqueName: \"kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.574756 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.574728    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.574817 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.574800    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.574865 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.574839    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.574937 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.574919    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.604557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.604533    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1e6fe1\"/\"distinct-w0-1e6fe1-kserve-self-signed-certs\""
Nov 03 20:47:00.668206 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.668147    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:00.675738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.675703    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5c8\" (UniqueName: \"kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.675888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.675745    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.675888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.675804    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.675888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.675835    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.675888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.675870    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.676091 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.676006    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.676177 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.676157    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.676239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.676220    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.676330 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.676309    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.678222 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.678193    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.678476 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.678462    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.734152 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.734123    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5c8\" (UniqueName: \"kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8\") pod \"distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") " pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:00.819455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:00.819368    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:01.152925 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:01.152889    5318 generic.go:356] "Generic (PLEG): container finished" podID="4721126a-d449-4123-9ac8-e8e215cd68f3" containerID="3bba59f376b82533024e5ed6ab561566bad0e7fe2e98913a1c2e9ae696f58e0d" exitCode=0
Nov 03 20:47:01.153078 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:01.152958    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66" event={"ID":"4721126a-d449-4123-9ac8-e8e215cd68f3","Type":"ContainerDied","Data":"3bba59f376b82533024e5ed6ab561566bad0e7fe2e98913a1c2e9ae696f58e0d"}
Nov 03 20:47:01.153078 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:01.153006    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66" event={"ID":"4721126a-d449-4123-9ac8-e8e215cd68f3","Type":"ContainerStarted","Data":"50486318b7c9cfe44f87ae1b1e3525981879a876c2a507c7e61dd97e24ad0e0f"}
Nov 03 20:47:01.194927 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:01.194904    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"]
Nov 03 20:47:01.207157 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:47:01.207130    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cc4d93_0a99_4379_b682_33fc3fed8524.slice/crio-6da0e4ebe04ecec2141ee3a258d7723b5df46a55d7f5c98df5a132626fc1f0db WatchSource:0}: Error finding container 6da0e4ebe04ecec2141ee3a258d7723b5df46a55d7f5c98df5a132626fc1f0db: Status 404 returned error can't find the container with id 6da0e4ebe04ecec2141ee3a258d7723b5df46a55d7f5c98df5a132626fc1f0db
Nov 03 20:47:02.160055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.160019    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" event={"ID":"55cc4d93-0a99-4379-b682-33fc3fed8524","Type":"ContainerStarted","Data":"d254e2d02671152ae92a00c192c70cc3ef87e31e438ae17390c0313b72c57c63"}
Nov 03 20:47:02.160055 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.160060    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" event={"ID":"55cc4d93-0a99-4379-b682-33fc3fed8524","Type":"ContainerStarted","Data":"6da0e4ebe04ecec2141ee3a258d7723b5df46a55d7f5c98df5a132626fc1f0db"}
Nov 03 20:47:02.298422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.298399    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:47:02.310671 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.310622    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podStartSLOduration=2.310606018 podStartE2EDuration="2.310606018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:02.310530539 +0000 UTC m=+1015.290087369" watchObservedRunningTime="2026-11-03 20:47:02.310606018 +0000 UTC m=+1015.290162859"
Nov 03 20:47:02.390983 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.390954    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs\" (UniqueName: \"kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs\") pod \"4721126a-d449-4123-9ac8-e8e215cd68f3\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") "
Nov 03 20:47:02.391197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.391007    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca\") pod \"4721126a-d449-4123-9ac8-e8e215cd68f3\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") "
Nov 03 20:47:02.391197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.391041    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca\") pod \"4721126a-d449-4123-9ac8-e8e215cd68f3\" (UID: \"4721126a-d449-4123-9ac8-e8e215cd68f3\") "
Nov 03 20:47:02.391453 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.391431    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca" pod "4721126a-d449-4123-9ac8-e8e215cd68f3" (UID: "4721126a-d449-4123-9ac8-e8e215cd68f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:02.393018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.392998    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca" pod "4721126a-d449-4123-9ac8-e8e215cd68f3" (UID: "4721126a-d449-4123-9ac8-e8e215cd68f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:02.393079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.393036    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs" pod "4721126a-d449-4123-9ac8-e8e215cd68f3" (UID: "4721126a-d449-4123-9ac8-e8e215cd68f3"). InnerVolumeSpecName "kube-api-access-kp4cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:02.492241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.492166    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:02.492241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.492191    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4721126a-d449-4123-9ac8-e8e215cd68f3-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:02.492241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.492200    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp4cs\" (UniqueName: \"kubernetes.io/projected/4721126a-d449-4123-9ac8-e8e215cd68f3-kube-api-access-kp4cs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:02.820481 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:02.820399    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:03.166931 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:03.166902    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"
Nov 03 20:47:03.167322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:03.166950    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66" event={"ID":"4721126a-d449-4123-9ac8-e8e215cd68f3","Type":"ContainerDied","Data":"50486318b7c9cfe44f87ae1b1e3525981879a876c2a507c7e61dd97e24ad0e0f"}
Nov 03 20:47:03.167322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:03.167007    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50486318b7c9cfe44f87ae1b1e3525981879a876c2a507c7e61dd97e24ad0e0f"
Nov 03 20:47:03.816545 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:03.816511    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:05.810296 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:05.810258    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:47:08.579856 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:08.579757    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:10.820409 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:10.820374    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:10.832820 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:10.832798    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:11.223457 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:11.223425    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:47:13.280194 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:13.280158    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:17.793265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:17.793227    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:18.534130 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:18.534094    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:22.383869 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:22.383826    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:25.684839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.684805    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"]
Nov 03 20:47:25.685494 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.685474    5318 cpu_manager.go:414] "containerMap: removing container" podUID="4721126a-d449-4123-9ac8-e8e215cd68f3" containerName="revoke-keys"
Nov 03 20:47:25.685619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.685494    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4721126a-d449-4123-9ac8-e8e215cd68f3" containerName="revoke-keys"
Nov 03 20:47:25.685619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.685607    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="4721126a-d449-4123-9ac8-e8e215cd68f3" containerName="revoke-keys"
Nov 03 20:47:25.688743 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.688724    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.737757 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.737715    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0cd49b-openshift-default-dockercfg-qtdr4\""
Nov 03 20:47:25.786172 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.786139    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"]
Nov 03 20:47:25.819928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.819891    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.819949    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.819996    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.820040    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820088 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.820078    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820311 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.820102    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820311 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.820225    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s477f\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820311 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.820288    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.820311 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.820307    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.921777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.921740    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s477f\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.921918 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.921820    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.921918 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.921856    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922009 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.921923    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922009 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.921995    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922111 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922019    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922111 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922078    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922128    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922166    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922331 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922211    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922331 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922245    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922507    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922634 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922580    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.922856 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.922835    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.924570 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.924549    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.925101 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.925059    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.937907 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:47:25.937845    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:47:25.979219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.979187    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s477f\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:25.979379 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:25.979190    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:26.001328 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:26.001305    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:26.424980 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:26.424940    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"]
Nov 03 20:47:26.459833 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:47:26.459799    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea2e484_0d8f_45c7_b03b_5037d4c5dae2.slice/crio-01ec05d5e633b4352f9547084c5fa0ff309d36b44940394827a5f62b21a2e6e9 WatchSource:0}: Error finding container 01ec05d5e633b4352f9547084c5fa0ff309d36b44940394827a5f62b21a2e6e9: Status 404 returned error can't find the container with id 01ec05d5e633b4352f9547084c5fa0ff309d36b44940394827a5f62b21a2e6e9
Nov 03 20:47:26.462119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:26.462090    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:26.462204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:26.462155    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:26.462204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:26.462183    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:27.008758 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:27.008724    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:27.303700 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:27.303605    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" event={"ID":"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2","Type":"ContainerStarted","Data":"032d268b87c520d358f9fe48910b737cafca94d96e40ddd81225944bf512f68b"}
Nov 03 20:47:27.303700 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:27.303667    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" event={"ID":"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2","Type":"ContainerStarted","Data":"01ec05d5e633b4352f9547084c5fa0ff309d36b44940394827a5f62b21a2e6e9"}
Nov 03 20:47:27.418866 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:27.418809    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podStartSLOduration=2.418793314 podStartE2EDuration="2.418793314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:27.41614452 +0000 UTC m=+1040.395701361" watchObservedRunningTime="2026-11-03 20:47:27.418793314 +0000 UTC m=+1040.398350158"
Nov 03 20:47:28.002065 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:28.002030    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:28.003979 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:28.003953    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:47:28.004073 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:28.004009    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:47:29.001867 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:29.001825    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:47:29.002398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:29.001882    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:47:30.002330 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:30.002291    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:47:30.002723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:30.002389    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:47:31.002317 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:31.002286    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:47:31.002505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:31.002358    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:47:31.514763 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:31.514729    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:32.002063 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:32.002027    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:47:32.002265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:32.002104    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:47:33.075680 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:33.075642    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:33.076160 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:33.076075    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:33.076840 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:33.076822    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:47:36.160727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:36.160694    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:37.734008 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:37.733978    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:47:38.869368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.869322    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"]
Nov 03 20:47:38.874442 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.874417    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.923165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.923143    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1e6fe1-openshift-default-dockercfg-gpj7x\""
Nov 03 20:47:38.974491 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974447    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974676 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974570    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974676 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974617    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974811 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974689    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974811 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974734    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974811 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974757    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hpz\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974962 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974829    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.974962 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.974877    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.975468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.975441    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:38.975866 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:38.975817    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"]
Nov 03 20:47:39.076522 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076488    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.076677 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076568    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.076677 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076612    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.076803 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076689    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.076803 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076718    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.076803 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076745    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hpz\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.076803 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076795    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.077012 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076844    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.077012 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.076877    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.077544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.077286    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.077544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.077413    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.077544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.077499    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.078047 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.078021    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.078624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.078600    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.079240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.079215    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.080068 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.080047    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.133976 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.133926    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.134181 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.134162    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hpz\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.188931 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.188904    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:39.613256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.613228    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"]
Nov 03 20:47:39.647720 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:47:39.647690    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e06b7a_a007_44bb_bc50_8e766d772f18.slice/crio-82fcc71c3d1139b1bfbf0b72257d0e899faea8c5861d9121cd4b4fc11794c406 WatchSource:0}: Error finding container 82fcc71c3d1139b1bfbf0b72257d0e899faea8c5861d9121cd4b4fc11794c406: Status 404 returned error can't find the container with id 82fcc71c3d1139b1bfbf0b72257d0e899faea8c5861d9121cd4b4fc11794c406
Nov 03 20:47:39.649955 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.649924    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:39.650019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.649990    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:39.650058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:39.650018    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:40.396751 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:40.396698    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" event={"ID":"05e06b7a-a007-44bb-bc50-8e766d772f18","Type":"ContainerStarted","Data":"c493d494ad26c1d2be8da87303693804abec49954cf8fbc84fd49fdfebaeae59"}
Nov 03 20:47:40.397173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:40.396770    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" event={"ID":"05e06b7a-a007-44bb-bc50-8e766d772f18","Type":"ContainerStarted","Data":"82fcc71c3d1139b1bfbf0b72257d0e899faea8c5861d9121cd4b4fc11794c406"}
Nov 03 20:47:40.510833 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:40.510780    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podStartSLOduration=2.510767238 podStartE2EDuration="2.510767238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:40.509417897 +0000 UTC m=+1053.488974736" watchObservedRunningTime="2026-11-03 20:47:40.510767238 +0000 UTC m=+1053.490324077"
Nov 03 20:47:40.685046 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:40.684789    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:41.190359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:41.189633    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:41.191995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:41.191966    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:41.192137 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:41.192010    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:47:42.190858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:42.190820    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:42.191390 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:42.190895    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:47:43.191361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:43.190702    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:43.191361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:43.190762    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:47:44.185361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:44.181801    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:44.189697 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:44.189668    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:44.189956 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:44.189929    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:47:45.190362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:45.189659    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:47:45.190362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:45.189719    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:47:45.290399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:45.290330    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:46.503302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:46.503273    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:46.503874 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:46.503794    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:46.504496 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:46.504473    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 20:47:49.870703 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:49.870649    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:54.294842 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.294806    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"]
Nov 03 20:47:54.299217 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.299192    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.345875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.345845    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpzw\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346024 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.345889    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346024 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.345944    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346160 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.346145    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346222 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.346196    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346278 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.346224    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346278 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.346253    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346492 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.346455    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.346532 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.346515    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.362769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.362731    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:54.363644 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.363621    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"]
Nov 03 20:47:54.447150 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447121    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447203    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447249    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447274    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447295    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447380    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447414    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447476    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpzw\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447504    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447568    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.447851 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.447802    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.448125 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.448098    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.448379 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.448351    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.448480 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.448364    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.450132 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.450101    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.450295 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.450273    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.505665 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.505579    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.505892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.505866    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpzw\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:54.595407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.595254    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:54.616473 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:54.615804    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:55.055984 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.055953    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"]
Nov 03 20:47:55.057998 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:47:55.057975    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df2746c_2881_4150_875d_e8aacbe6f90b.slice/crio-92bc254f0338ee7472fa79b7fd741f63cd76ab39d02c38770e626337300491d1 WatchSource:0}: Error finding container 92bc254f0338ee7472fa79b7fd741f63cd76ab39d02c38770e626337300491d1: Status 404 returned error can't find the container with id 92bc254f0338ee7472fa79b7fd741f63cd76ab39d02c38770e626337300491d1
Nov 03 20:47:55.060603 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.060569    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:55.060692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.060635    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:55.060692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.060664    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:55.207181 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.207151    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"]
Nov 03 20:47:55.211544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.211522    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259067    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259216    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259248    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259416    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259476    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259550    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259599    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259643    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97q6\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.259678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.259661    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.261246 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.261227    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-504eb0-openshift-default-dockercfg-4ckhb\""
Nov 03 20:47:55.311089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.311013    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"]
Nov 03 20:47:55.361033 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361004    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361245 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361221    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361295 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361272    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361349 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361321    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361349 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361321    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361456 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361434    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97q6\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361510 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361472    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361566 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361546    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361595    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361612    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361624    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.361958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361933    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.362134 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.361988    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.362244 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.362217    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.363722 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.363691    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.363827 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.363809    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.418739 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.418718    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.418951 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.418929    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97q6\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.479698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.479662    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" event={"ID":"1df2746c-2881-4150-875d-e8aacbe6f90b","Type":"ContainerStarted","Data":"fae47d652f35ecd8eaf56b054c7e00d7a469cddac5ea9f0b7ebb2e3c14c82135"}
Nov 03 20:47:55.479844 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.479729    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" event={"ID":"1df2746c-2881-4150-875d-e8aacbe6f90b","Type":"ContainerStarted","Data":"92bc254f0338ee7472fa79b7fd741f63cd76ab39d02c38770e626337300491d1"}
Nov 03 20:47:55.526092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.526060    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:55.595890 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.595802    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podStartSLOduration=1.5957883179999999 podStartE2EDuration="1.595788318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:55.592839409 +0000 UTC m=+1068.572396288" watchObservedRunningTime="2026-11-03 20:47:55.595788318 +0000 UTC m=+1068.575345158"
Nov 03 20:47:55.616311 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.616247    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:47:55.934322 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:47:55.934286    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:47:55.959995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.959975    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"]
Nov 03 20:47:55.961813 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:47:55.961783    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f6e609_748a_4985_80af_eee106e9158a.slice/crio-e1a11e17d221f84b48d4b6ac2079d68d52b0e39b19a50b3bf384a6b37e34e898 WatchSource:0}: Error finding container e1a11e17d221f84b48d4b6ac2079d68d52b0e39b19a50b3bf384a6b37e34e898: Status 404 returned error can't find the container with id e1a11e17d221f84b48d4b6ac2079d68d52b0e39b19a50b3bf384a6b37e34e898
Nov 03 20:47:55.964286 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.964248    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:55.964425 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.964329    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:55.964425 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:55.964412    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:47:56.499946 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:56.499913    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" event={"ID":"85f6e609-748a-4985-80af-eee106e9158a","Type":"ContainerStarted","Data":"95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb"}
Nov 03 20:47:56.500511 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:56.500490    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" event={"ID":"85f6e609-748a-4985-80af-eee106e9158a","Type":"ContainerStarted","Data":"e1a11e17d221f84b48d4b6ac2079d68d52b0e39b19a50b3bf384a6b37e34e898"}
Nov 03 20:47:56.526850 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:56.526826    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:47:56.613199 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:56.613135    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podStartSLOduration=1.613089241 podStartE2EDuration="1.613089241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:56.612443959 +0000 UTC m=+1069.592000800" watchObservedRunningTime="2026-11-03 20:47:56.613089241 +0000 UTC m=+1069.592646084"
Nov 03 20:47:56.617731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:56.617689    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:56.617868 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:56.617762    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:47:57.535726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:57.535624    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:47:57.535726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:57.535672    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:47:57.627373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:57.627019    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:57.627373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:57.627095    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:47:58.527410 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:58.527382    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:47:58.527583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:58.527430    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:47:58.616518 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:58.616479    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:58.617092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:58.617061    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:47:59.000362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:59.000207    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:59.526729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:59.526686    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:47:59.526900 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:59.526751    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:47:59.617469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:59.616986    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:47:59.617469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:47:59.617056    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:48:00.526737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:00.526697    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:48:00.526912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:00.526748    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:48:00.617013 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:00.616858    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:48:00.617013 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:00.616952    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:48:01.526377 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:01.526322    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:48:01.526888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:01.526403    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:48:01.616647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:01.616611    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:48:01.616830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:01.616680    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:48:02.526730 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:02.526693    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:48:02.527163 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:02.526766    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:48:02.622681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:02.622650    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:48:02.623181 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:02.623139    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:48:02.624053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:02.624025    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 20:48:03.493063 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:03.493033    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:03.532309 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:03.532281    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:48:03.533116 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:03.533091    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:48:03.533992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:03.533973    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:48:07.102227 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:07.100365    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:08.173353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:08.173312    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:12.704710 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:12.704667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:14.931438 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:14.931405    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:48:16.801533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:16.801410    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:17.332669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:17.332634    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:18.871782 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:18.871750    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:21.798827 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:21.798796    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:25.933482 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:48:25.933437    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:48:26.358312 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:26.358281    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:30.349796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:30.349758    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:30.923624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:30.923580    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:35.571180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:35.571143    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:39.835813 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.835783    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"]
Nov 03 20:48:39.840122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.840098    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.888834 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.888808    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3b8feb-openshift-default-dockercfg-q9mnd\""
Nov 03 20:48:39.937938 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.937913    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"]
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.947520    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5lnb\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.947921    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.947971    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.948006    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.948085    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.948120    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.948140    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.948177    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:39.948350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:39.948205    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.049683 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049649    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.049861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049690    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.049861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049745    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.049861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049780    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.049861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049820    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5lnb\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.049861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049849    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049872    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.049909    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050122 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.050031    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050284 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.050157    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050284 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.050224    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.050310    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.050433    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.050681 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.050661    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.052206 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.052175    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.052303 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.052286    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.072873 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.072820    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:40.107265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.107235    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5lnb\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.107421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.107247    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.158595 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.158555    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:40.586858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.586823    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"]
Nov 03 20:48:40.588594 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:48:40.588566    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7da5416_8610_47b5_8b47_92028576c2ad.slice/crio-3dc80230b1709328ef7605ae2325292eab6dbf1224e5c02dba891f4c2455c4de WatchSource:0}: Error finding container 3dc80230b1709328ef7605ae2325292eab6dbf1224e5c02dba891f4c2455c4de: Status 404 returned error can't find the container with id 3dc80230b1709328ef7605ae2325292eab6dbf1224e5c02dba891f4c2455c4de
Nov 03 20:48:40.591489 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.591457    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:40.591598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.591540    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:40.591598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.591589    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:40.792089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.792053    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" event={"ID":"d7da5416-8610-47b5-8b47-92028576c2ad","Type":"ContainerStarted","Data":"87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107"}
Nov 03 20:48:40.792248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.792121    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" event={"ID":"d7da5416-8610-47b5-8b47-92028576c2ad","Type":"ContainerStarted","Data":"3dc80230b1709328ef7605ae2325292eab6dbf1224e5c02dba891f4c2455c4de"}
Nov 03 20:48:40.914918 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:40.914856    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podStartSLOduration=1.9148367309999998 podStartE2EDuration="1.914836731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:40.913912263 +0000 UTC m=+1113.893469139" watchObservedRunningTime="2026-11-03 20:48:40.914836731 +0000 UTC m=+1113.894393573"
Nov 03 20:48:41.164360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:41.164144    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:41.164360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:41.164311    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:41.164666 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:41.164388    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:42.159920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.159883    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:42.160412 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.159942    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:42.252648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.252616    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"]
Nov 03 20:48:42.256559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.256527    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.306411 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.306389    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"kube-root-ca.crt\""
Nov 03 20:48:42.308706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.308680    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"openshift-service-ca.crt\""
Nov 03 20:48:42.352417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.352391    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"default-dockercfg-56s89\""
Nov 03 20:48:42.352417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.352422    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"distinct-w6-504eb0-kserve-self-signed-certs\""
Nov 03 20:48:42.354936 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.354914    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"]
Nov 03 20:48:42.394776 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.394745    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.394933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.394796    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.394933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.394813    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.395034 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.394947    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5454\" (UniqueName: \"kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.395034 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.394991    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.395116 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.395084    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.407051 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.407021    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:42.496230 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496134    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5454\" (UniqueName: \"kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496428 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496230    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496428 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496300    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496428 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496403    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496603 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496445    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496603 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496468    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496871 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496841    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496941 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496887    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.496941 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.496901    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.498735 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.498710    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.498878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.498860    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.553500 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.553462    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5454\" (UniqueName: \"kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454\") pod \"distinct-w6-504eb0-kserve-79969b455f-8zhpb\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:42.598358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:42.598216    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:43.011147 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:43.011122    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"]
Nov 03 20:48:43.013896 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:48:43.013868    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod459be02a_5217_424b_bf75_65009ee0e706.slice/crio-7362df59a96eb7732848ab2dda2a1e360c368c16caf8a47815776ce5f58c1134 WatchSource:0}: Error finding container 7362df59a96eb7732848ab2dda2a1e360c368c16caf8a47815776ce5f58c1134: Status 404 returned error can't find the container with id 7362df59a96eb7732848ab2dda2a1e360c368c16caf8a47815776ce5f58c1134
Nov 03 20:48:43.160410 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:43.160376    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:43.160853 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:43.160436    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:43.828513 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:43.828461    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" event={"ID":"459be02a-5217-424b-bf75-65009ee0e706","Type":"ContainerStarted","Data":"919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137"}
Nov 03 20:48:43.828513 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:43.828527    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" event={"ID":"459be02a-5217-424b-bf75-65009ee0e706","Type":"ContainerStarted","Data":"7362df59a96eb7732848ab2dda2a1e360c368c16caf8a47815776ce5f58c1134"}
Nov 03 20:48:43.941755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:43.941673    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podStartSLOduration=1.941657081 podStartE2EDuration="1.941657081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:43.940763284 +0000 UTC m=+1116.920320123" watchObservedRunningTime="2026-11-03 20:48:43.941657081 +0000 UTC m=+1116.921213924"
Nov 03 20:48:44.159944 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:44.159909    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:44.160092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:44.159967    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:44.598545 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:44.598512    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:44.789360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:44.786840    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:45.159569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:45.159538    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:45.159755 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:45.159599    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:46.159870 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:46.159834    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:46.160326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:46.159912    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:47.160432 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:47.160395    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:48:47.160812 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:47.160457    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:48:48.164512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:48.164476    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:48.164936 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:48.164909    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:48.165686 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:48.165669    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 20:48:49.371178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:49.371139    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:49.752992 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:49.752921    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:50.737309 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:50.737279    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:52.598574 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:52.598534    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:52.611247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:52.611218    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:52.890848 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:52.890763    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:48:53.819537 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:53.819500    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:55.933449 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:48:55.933413    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:48:58.274612 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:58.274537    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:58.477405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:48:58.477367    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:02.871145 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:02.871108    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:03.419907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:03.419872    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:49:07.425546 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:07.425496    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:08.680278 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:08.680234    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:10.491345 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.491298    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"]
Nov 03 20:49:10.495119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.495082    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.532713 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.532680    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.532854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.532761    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.532854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.532819    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt8d\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.532944 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.532871    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.532944 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.532893    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.533043 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.533017    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.533100 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.533066    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.533156 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.533097    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.533210 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.533185    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.543802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.543777    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3d2008-openshift-default-dockercfg-z445t\""
Nov 03 20:49:10.593937 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.593908    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"]
Nov 03 20:49:10.634040 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634016    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634050    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634076    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634103    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634154    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634263    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634289    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt8d\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634368    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634444 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634401    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634667 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634475    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634667 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634532    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634667 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634590    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.634805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.634737    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.635086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.635063    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.636572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.636549    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.636745 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.636730    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.690900 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.690879    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.691051 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.691031    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt8d\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-4557x\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:10.808305 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:10.808244    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:11.228398 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:11.227450    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"]
Nov 03 20:49:11.234554 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:11.234522    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:11.234636 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:11.234583    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:11.234636 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:11.234611    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:49:11.977918 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:11.977879    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:12.008225 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:12.008186    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" event={"ID":"c6b6e8ce-6f19-479a-8ab4-38368e5b878c","Type":"ContainerStarted","Data":"66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462"}
Nov 03 20:49:12.008415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:12.008251    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" event={"ID":"c6b6e8ce-6f19-479a-8ab4-38368e5b878c","Type":"ContainerStarted","Data":"5a3d72fb343027c23fc45f78627028afa1ba809a2eb7d10dfe5ac4b06075a5e2"}
Nov 03 20:49:12.122766 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:12.122714    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podStartSLOduration=2.122699989 podStartE2EDuration="2.122699989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:12.120434608 +0000 UTC m=+1145.099991448" watchObservedRunningTime="2026-11-03 20:49:12.122699989 +0000 UTC m=+1145.102256902"
Nov 03 20:49:12.808418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:12.808389    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:12.810104 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:12.810071    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:49:12.810235 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:12.810118    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:49:13.808689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:13.808653    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:49:13.809240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:13.808719    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:49:14.605598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:14.605561    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:14.800201 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:14.800171    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:14.808789 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:14.808762    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:49:14.809237 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:14.808811    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:49:15.809776 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:15.809732    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:49:15.810263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:15.809791    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:49:16.488357 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:16.485929    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:16.809037 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:16.808948    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:49:16.809288 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:16.809260    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:49:18.143570 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:18.143535    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:18.144437 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:18.144409    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:18.145200 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:18.145179    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 20:49:19.197875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:19.197814    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:21.154640 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:21.154607    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:25.725257 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:25.725222    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:25.934280 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:49:25.934243    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:49:30.270079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:30.270040    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:34.802581 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:34.802545    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:39.367714 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:39.367674    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:43.618877 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:43.618834    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:43.741516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:43.741478    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:49:43.870128 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:43.870041    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:48.479657 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:48.479576    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:52.987368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:52.987313    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:55.933451 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:49:55.933402    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:49:57.483384 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:49:57.483329    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:01.966130 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:01.966096    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:06.514658 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:06.514596    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:07.449792 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:07.449766    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:50:07.455666 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:07.455648    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:50:11.064617 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:11.064577    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:14.558448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:14.558410    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:50:15.655229 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:15.655185    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:19.526361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:19.526311    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:20.057663 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:20.057632    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:22.045697 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:22.045657    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"]
Nov 03 20:50:22.091954 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:22.091924    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d801c4-wcn6n"]
Nov 03 20:50:23.008123 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:23.008083    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:23.290606 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:23.290514    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdca8312-e27b-409d-a83b-58895ef422de" path="/var/lib/kubelet/pods/cdca8312-e27b-409d-a83b-58895ef422de/volumes"
Nov 03 20:50:24.680850 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:24.680810    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:25.933155 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:50:25.933121    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:50:29.257933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:29.257905    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:33.784489 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:33.784455    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:36.861854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:36.861821    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"]
Nov 03 20:50:36.865410 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:36.865378    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:36.960097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:36.960060    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3d2008\"/\"distinct-2-w4-3d2008-kserve-self-signed-certs\""
Nov 03 20:50:36.962539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:36.962513    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"]
Nov 03 20:50:37.010091 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.010057    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.010292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.010148    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4m29\" (UniqueName: \"kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.010292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.010189    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.010292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.010270    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.010512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.010297    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.010512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.010398    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.018999 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.018970    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:37.111208 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111166    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111467 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111284    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111467 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111386    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4m29\" (UniqueName: \"kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111467 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111428    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111496    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111521    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111669    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111827 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111764    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.111910 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.111856    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.114664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.114630    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.115176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.115143    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.183167 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.183091    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4m29\" (UniqueName: \"kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29\") pod \"distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") " pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.478690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.478608    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:37.852440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:37.852417    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"]
Nov 03 20:50:37.854501 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:50:37.854464    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c9222a3_7d8e_474d_8f67_04804f48aaac.slice/crio-97115e0b0d4a82d35e6dddae984754a554b00a0f6bb2ea43bdb868d76bcf6612 WatchSource:0}: Error finding container 97115e0b0d4a82d35e6dddae984754a554b00a0f6bb2ea43bdb868d76bcf6612: Status 404 returned error can't find the container with id 97115e0b0d4a82d35e6dddae984754a554b00a0f6bb2ea43bdb868d76bcf6612
Nov 03 20:50:38.369448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:38.369418    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:38.526134 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:38.526097    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" event={"ID":"0c9222a3-7d8e-474d-8f67-04804f48aaac","Type":"ContainerStarted","Data":"1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478"}
Nov 03 20:50:38.526306 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:38.526145    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" event={"ID":"0c9222a3-7d8e-474d-8f67-04804f48aaac","Type":"ContainerStarted","Data":"97115e0b0d4a82d35e6dddae984754a554b00a0f6bb2ea43bdb868d76bcf6612"}
Nov 03 20:50:38.636605 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:38.636511    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podStartSLOduration=2.636497574 podStartE2EDuration="2.636497574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:38.635208083 +0000 UTC m=+1231.614764924" watchObservedRunningTime="2026-11-03 20:50:38.636497574 +0000 UTC m=+1231.616054413"
Nov 03 20:50:39.478901 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:39.478862    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:40.672253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:40.672214    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:42.870283 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:42.870244    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:47.478928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:47.478889    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:47.498609 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:47.498579    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:47.503242 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:47.503194    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:47.599920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:47.599890    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:50:49.201989 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.201947    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"]
Nov 03 20:50:49.205929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.205900    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.301137 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.301101    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"distinct-2-w6-504eb0-kserve-self-signed-certs\""
Nov 03 20:50:49.303616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.303589    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"]
Nov 03 20:50:49.337200 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.337167    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.337373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.337220    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.337373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.337315    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.337506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.337430    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.337506 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.337486    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.337612 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.337596    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.370835 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.370791    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:49.438665 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.438624    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.438839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.438700    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.438839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.438761    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.438839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.438797    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.439007 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.438847    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.439007 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.438896    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.439161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.439126    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.439294 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.439266    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.439548 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.439524    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.441812 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.441783    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.441943 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.441925    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.496585 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.496523    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r\") pod \"distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") " pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.519351 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.519307    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:49.897227 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:49.897201    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"]
Nov 03 20:50:49.899135 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:50:49.899103    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cf6b6eb_dcdf_4f67_8612_ad72762077e0.slice/crio-46f27991d229da7fdfc20420d56b9a55419ee167d0acb16fefb153d3f71db9aa WatchSource:0}: Error finding container 46f27991d229da7fdfc20420d56b9a55419ee167d0acb16fefb153d3f71db9aa: Status 404 returned error can't find the container with id 46f27991d229da7fdfc20420d56b9a55419ee167d0acb16fefb153d3f71db9aa
Nov 03 20:50:50.637371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:50.637321    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" event={"ID":"0cf6b6eb-dcdf-4f67-8612-ad72762077e0","Type":"ContainerStarted","Data":"2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6"}
Nov 03 20:50:50.637726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:50.637385    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" event={"ID":"0cf6b6eb-dcdf-4f67-8612-ad72762077e0","Type":"ContainerStarted","Data":"46f27991d229da7fdfc20420d56b9a55419ee167d0acb16fefb153d3f71db9aa"}
Nov 03 20:50:50.749028 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:50.748984    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podStartSLOduration=1.7489707650000001 podStartE2EDuration="1.748970765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:50.746913668 +0000 UTC m=+1243.726470510" watchObservedRunningTime="2026-11-03 20:50:50.748970765 +0000 UTC m=+1243.728527637"
Nov 03 20:50:51.519668 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:51.519635    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:52.120442 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:52.120405    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:55.932045 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:50:55.932013    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:50:56.653780 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:56.653749    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:59.520049 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:59.520009    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:59.532531 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:59.532509    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:59.707273 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:59.707235    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:59.709529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:59.709500    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:50:59.906074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:50:59.906042    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:01.169241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:01.169203    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:02.038091 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:02.038056    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:51:06.072970 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:06.072939    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:10.973633 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:10.973597    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:13.869240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:13.869181    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:15.511618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:15.511529    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:19.059590 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:19.059549    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:19.863347 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:19.863305    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:20.065609 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:20.065577    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:24.581586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:24.581544    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:25.612697 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.612664    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"]
Nov 03 20:51:25.616650 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.616621    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.659230 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.659206    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"]
Nov 03 20:51:25.663078 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.663055    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.709704 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.709674    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"]
Nov 03 20:51:25.755218 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755176    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.755400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755236    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755298    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755364    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.755584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755453    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755478    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.755584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755512    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755563    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755634    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdgw\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755682    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.755802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755743    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.755802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755763    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.756009 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755853    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.756009 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755893    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.756009 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.755940    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phx5f\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.756168 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.756013    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.756168 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.756043    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.756168 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.756078    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.787573 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.787546    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"]
Nov 03 20:51:25.856689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856658    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdgw\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.856875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856703    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.856875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856727    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.856875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856754    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.856875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856813    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.856875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856839    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.857140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856884    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phx5f\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856925    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856956    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.857140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.856988    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857066    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857110    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.857892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857148    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.857892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857214    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857272    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.857892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857347    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857628    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.857892 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.857685    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.858259 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.858116    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.859905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.858545    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.859905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.858798    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.859905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.858868    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.859905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.858895    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.859905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.859170    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.859905 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.859775    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.860964 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.860937    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.861114 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.861092    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.861283 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.861262    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.861541 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.861521    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.862002 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.861978    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.863809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.863758    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.864296 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.864275    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.914677 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.914637    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.914677 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.914659    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phx5f\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:25.915173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.915150    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdgw\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.915721 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.915700    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.931617 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.931590    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:25.935844 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:51:25.935814    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:51:25.976858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:25.976830    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:26.352425 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.352402    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"]
Nov 03 20:51:26.353397 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:51:26.353373    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9156fa97_8691_42d5_bc3e_1b4b0ad3db1c.slice/crio-d65ba469fde91f669c1a89a6a987e84a75d91de27307f9188655aee33f271f6e WatchSource:0}: Error finding container d65ba469fde91f669c1a89a6a987e84a75d91de27307f9188655aee33f271f6e: Status 404 returned error can't find the container with id d65ba469fde91f669c1a89a6a987e84a75d91de27307f9188655aee33f271f6e
Nov 03 20:51:26.356023 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.355985    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.356107 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.356071    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.356154 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.356104    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.400026 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.400005    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"]
Nov 03 20:51:26.403599 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:51:26.403567    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b15112_2cf7_4c6b_9900_8dd57239041b.slice/crio-be97bbd5356420a2fa59ee3b8aee348788bd995f89554b6c20a32c52bcbc6a7e WatchSource:0}: Error finding container be97bbd5356420a2fa59ee3b8aee348788bd995f89554b6c20a32c52bcbc6a7e: Status 404 returned error can't find the container with id be97bbd5356420a2fa59ee3b8aee348788bd995f89554b6c20a32c52bcbc6a7e
Nov 03 20:51:26.406714 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.406677    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.406799 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.406759    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.406839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.406802    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:26.500725 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.500695    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"]
Nov 03 20:51:26.505275 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.505244    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.571539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.569698    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.575727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.569740    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.575727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575504    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.575727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575553    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.575727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575603    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.575727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575630    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.575727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575671    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.576085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575745    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4pmt\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.576085 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.575840    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.579708 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.579681    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"]
Nov 03 20:51:26.677178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677103    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677170    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677198    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677277    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677320    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677384    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677415    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677454    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677532    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4pmt\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677588    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677649    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.677746 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.677697    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.678256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.678155    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.678431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.678405    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.680022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.679994    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.680184 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.680167    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.734630 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.734599    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.735091 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.735068    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4pmt\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.822284 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.822243    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:26.910893 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.910856    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" event={"ID":"f9b15112-2cf7-4c6b-9900-8dd57239041b","Type":"ContainerStarted","Data":"e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed"}
Nov 03 20:51:26.911039 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.910927    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" event={"ID":"f9b15112-2cf7-4c6b-9900-8dd57239041b","Type":"ContainerStarted","Data":"be97bbd5356420a2fa59ee3b8aee348788bd995f89554b6c20a32c52bcbc6a7e"}
Nov 03 20:51:26.914582 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.914549    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" event={"ID":"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c","Type":"ContainerStarted","Data":"0a320201155b7f0575a00e7ca5cf1f45678631e4230e47e94b0f6718436fc733"}
Nov 03 20:51:26.914723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.914595    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" event={"ID":"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c","Type":"ContainerStarted","Data":"d65ba469fde91f669c1a89a6a987e84a75d91de27307f9188655aee33f271f6e"}
Nov 03 20:51:26.939923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.939839    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:26.991745 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:26.989452    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:27.034388 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.033766    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podStartSLOduration=2.033746916 podStartE2EDuration="2.033746916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:27.030448241 +0000 UTC m=+1280.010005093" watchObservedRunningTime="2026-11-03 20:51:27.033746916 +0000 UTC m=+1280.013303755"
Nov 03 20:51:27.155633 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.155564    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podStartSLOduration=2.155546919 podStartE2EDuration="2.155546919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:27.154780771 +0000 UTC m=+1280.134337705" watchObservedRunningTime="2026-11-03 20:51:27.155546919 +0000 UTC m=+1280.135103760"
Nov 03 20:51:27.262441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.261568    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"]
Nov 03 20:51:27.264353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.264297    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:27.264468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.264396    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:27.264468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.264441    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:27.941604 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.939840    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:51:27.941604 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.939901    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:51:27.946021 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.945175    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" event={"ID":"84a6ac6d-0255-4ca4-b53d-526371f74f93","Type":"ContainerStarted","Data":"e5ed65a2afda5a16ae334e86dacc961ca274bff988964a07043d0ab08f17d821"}
Nov 03 20:51:27.946021 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.945264    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" event={"ID":"84a6ac6d-0255-4ca4-b53d-526371f74f93","Type":"ContainerStarted","Data":"88f846949a1ab79cc66f8f30ed62a2f82ca75784ed43f779f3f701cf8c8d2fe1"}
Nov 03 20:51:27.978988 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.978868    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:51:27.978988 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:27.978925    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:51:28.063629 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.063565    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podStartSLOduration=2.063546038 podStartE2EDuration="2.063546038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.061675751 +0000 UTC m=+1281.041232626" watchObservedRunningTime="2026-11-03 20:51:28.063546038 +0000 UTC m=+1281.043102882"
Nov 03 20:51:28.826360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.825959    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:28.826360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.826120    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:51:28.826360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.826166    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:51:28.934358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.933973    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:51:28.934358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.934054    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:51:28.981361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.980764    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:51:28.981361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:28.980827    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:51:29.518367 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.518087    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:29.830416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.828712    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.830416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.828775    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:51:29.936761 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.936723    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.937229 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.937198    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:51:29.994360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.982680    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:51:29.994360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:29.982732    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:51:30.853986 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:30.847970    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.853986 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:30.848029    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:51:30.933371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:30.932294    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:51:30.933371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:30.932375    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:51:30.983773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:30.983735    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.005454 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.005408    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:51:31.824363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.823825    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.824363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.823905    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:51:31.946362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.938497    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.946362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.938559    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:51:31.980692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.980659    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:51:31.980940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:31.980914    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:51:32.822875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:32.822837    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.823262 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:32.822906    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:51:32.932276 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:32.932236    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.932485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:32.932310    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:51:32.984362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:32.980741    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:51:32.984362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:32.980803    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:51:33.823023 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:33.822988    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:51:33.823682 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:33.823632    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:51:34.061858 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.061828    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:34.062479 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.062447    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:34.063614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.063594    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 20:51:34.195962 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.195930    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:34.196639 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.196614    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:34.198049 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.198027    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 20:51:34.385252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:34.385220    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:35.229528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:35.229484    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:35.230805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:35.230699    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:35.231924 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:35.231906    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 20:51:38.971630 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:38.971596    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:41.701490 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:41.701456    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:43.820814 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:43.820778    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:48.305878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:48.305836    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:52.808165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:52.808117    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:55.933909 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:51:55.933873    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:51:57.478907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:57.478809    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:59.223836 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:59.223789    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:59.430246 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:51:59.430206    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:02.087571 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:02.087530    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:03.053504 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:03.053463    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"]
Nov 03 20:52:03.099379 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:03.099346    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8d05cb2-wqk66"]
Nov 03 20:52:03.255716 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:03.255682    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4721126a-d449-4123-9ac8-e8e215cd68f3" path="/var/lib/kubelet/pods/4721126a-d449-4123-9ac8-e8e215cd68f3/volumes"
Nov 03 20:52:06.677578 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:06.677542    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:07.423041 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:07.423010    5318 scope.go:122] "RemoveContainer" containerID="22630703e39bac393239caa2846ef6d5047fe1dad5fef68eadbf08134d7b84c0"
Nov 03 20:52:10.848460 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:10.848424    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:52:11.275549 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.275468    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:11.498121 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.498088    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"]
Nov 03 20:52:11.503886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.503859    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.544385 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.544311    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"]
Nov 03 20:52:11.548826 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.548805    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.590806 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.590781    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"]
Nov 03 20:52:11.612650 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.612620    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"]
Nov 03 20:52:11.625032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625004    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625115    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625147    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625208    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc25c\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625239    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625270    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625488 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625367    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29wf\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625488 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625404    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625488 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625431    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625494    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625538    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625573    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625617    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625647 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625637    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.625907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625662    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625708    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625768    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.625907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.625797    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.727018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.726960    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.727538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.727515    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.728178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.728093    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.728282 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.728252    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.728405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.728379    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f29wf\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.728515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.728457    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.728900 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.728491    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.729069 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.729050    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.729294 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.729254    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.729502 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.729113    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.729586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.729513    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.729689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.729673    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.729818 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.729800    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.730239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.730212    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.730505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.730287    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.730505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.730130    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.730903 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.730860    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.731097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.731015    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.731503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.731207    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.731684 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.731660    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.732095 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732076    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.732302 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732285    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.732421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.731882    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.732518 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732401    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.732678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732659    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.732799 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732784    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.732923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732908    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc25c\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.733082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.732968    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.733392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.733373    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.733533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.733451    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.733694 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.733674    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.733894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.733878    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.785869 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.785830    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29wf\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.789052 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.789024    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.789174 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.789134    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.789638 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.789615    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc25c\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:11.818090 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.818035    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:11.863309 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:11.863274    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:12.249355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.248388    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"]
Nov 03 20:52:12.262126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.260824    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.262126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.260901    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.262126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.260943    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.295986 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.295944    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"]
Nov 03 20:52:12.304719 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.304684    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.304989 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.304964    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.305176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.305143    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:12.412234 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.412147    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" event={"ID":"b8a818e0-b22b-48c7-ae0d-5db2310cc576","Type":"ContainerStarted","Data":"3589733447b17d4cdde37040c7dfc9824b4f464a9aacecff3bf0fe1c478543e9"}
Nov 03 20:52:12.417253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.416197    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" event={"ID":"e7453bc9-7604-4959-b822-acbf65383db8","Type":"ContainerStarted","Data":"cd15d080c8ed6a75014df19844c7d70e5fe323c575c101b7bf6edb80c0ff6d38"}
Nov 03 20:52:12.417253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.416252    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" event={"ID":"e7453bc9-7604-4959-b822-acbf65383db8","Type":"ContainerStarted","Data":"055e0a7ccfc75607c200bcbca13ea1ce99f6ee06a361d911844887f3f0be9599"}
Nov 03 20:52:12.611352 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.607438    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podStartSLOduration=1.607385802 podStartE2EDuration="1.607385802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:12.565688423 +0000 UTC m=+1325.545245256" watchObservedRunningTime="2026-11-03 20:52:12.607385802 +0000 UTC m=+1325.586942643"
Nov 03 20:52:12.818557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:12.818523    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:13.504362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.503463    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" event={"ID":"b8a818e0-b22b-48c7-ae0d-5db2310cc576","Type":"ContainerStarted","Data":"9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5"}
Nov 03 20:52:13.680724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.680659    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podStartSLOduration=2.6806374760000002 podStartE2EDuration="2.680637476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:13.666929971 +0000 UTC m=+1326.646486848" watchObservedRunningTime="2026-11-03 20:52:13.680637476 +0000 UTC m=+1326.660194316"
Nov 03 20:52:13.824417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.823449    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:52:13.824417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.823519    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:52:13.867361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.867087    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:13.867361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.867259    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:13.867361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:13.867305    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:14.819801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:14.819765    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:52:14.820427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:14.820385    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:52:14.864140 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:14.864102    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:14.864306 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:14.864227    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:15.827360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:15.820730    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:52:15.827360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:15.820810    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:52:15.864405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:15.864371    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:15.864569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:15.864427    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:16.431360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:16.430586    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:16.818874 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:16.818776    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:52:16.819142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:16.819114    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:52:16.864554 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:16.864521    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:16.864936 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:16.864574    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:17.835870 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:17.835830    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:52:17.836160 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:17.836120    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:52:17.863920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:17.863887    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:17.864100 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:17.863948    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:18.822361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:18.821494    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:52:18.822361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:18.821555    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:52:18.863891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:18.863857    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:18.864063 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:18.863912    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:19.864080 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:19.864033    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:52:19.864728 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:19.864117    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:52:20.253709 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.253610    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:20.268298 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.267491    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:20.268298 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.268231    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:20.269106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.269058    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 20:52:20.867977 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.867948    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:20.868474 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.868374    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:20.869007 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:20.868988    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:52:21.077925 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:21.077892    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:25.934701 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:52:25.934659    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:52:28.323607 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:28.323562    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:30.391483 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:30.391443    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:34.875310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:34.875250    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:38.938115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:38.938078    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:39.161534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:39.161502    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:39.428895 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:39.428861    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:42.601534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.598971    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"]
Nov 03 20:52:42.606282 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.606243    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.658400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.658371    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2a5ec9-openshift-default-dockercfg-kln2c\""
Nov 03 20:52:42.697692 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.697654    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.697906 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.697805    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.697978 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.697920    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.697978 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.697950    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.698142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.698031    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.698142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.698084    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.698248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.698149    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.698248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.698178    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.698248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.698230    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwm4j\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.712013 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.711961    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"]
Nov 03 20:52:42.799741 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799701    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.799741 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799744    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.799995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799774    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.799995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799824    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.799995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799843    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.799995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799890    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwm4j\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.799995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.799973    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.800258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.800067    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.800258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.800160    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.800258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.800170    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.800923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.800894    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.801047 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.800968    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.801047 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.800990    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.801288 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.801257    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.802222 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.802199    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.802517 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.802500    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.858248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.858182    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.858384 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.858325    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwm4j\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:42.923553 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:42.923525    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:43.348786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.348762    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"]
Nov 03 20:52:43.350676 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:52:43.350639    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d105468_e95b_484c_8592_d6b84aa21d63.slice/crio-fb703808a282e601d498c252b5431dc420bc5c8a2ecd4bf433a5b8eaea1b99eb WatchSource:0}: Error finding container fb703808a282e601d498c252b5431dc420bc5c8a2ecd4bf433a5b8eaea1b99eb: Status 404 returned error can't find the container with id fb703808a282e601d498c252b5431dc420bc5c8a2ecd4bf433a5b8eaea1b99eb
Nov 03 20:52:43.353176 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.353145    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:43.353265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.353210    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:43.353265 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.353242    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:43.733403 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.733297    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" event={"ID":"6d105468-e95b-484c-8592-d6b84aa21d63","Type":"ContainerStarted","Data":"3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b"}
Nov 03 20:52:43.733863 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.733406    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" event={"ID":"6d105468-e95b-484c-8592-d6b84aa21d63","Type":"ContainerStarted","Data":"fb703808a282e601d498c252b5431dc420bc5c8a2ecd4bf433a5b8eaea1b99eb"}
Nov 03 20:52:43.849018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.848967    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podStartSLOduration=1.848952981 podStartE2EDuration="1.848952981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:43.846302782 +0000 UTC m=+1356.825859619" watchObservedRunningTime="2026-11-03 20:52:43.848952981 +0000 UTC m=+1356.828509820"
Nov 03 20:52:43.923914 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:43.923875    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:44.091365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:44.090297    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:44.925278 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:44.925241    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:44.925771 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:44.925300    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:45.931995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:45.926789    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:45.931995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:45.926850    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:46.927812 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:46.927775    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:46.928102 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:46.928073    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:47.461371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.459429    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.591363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.589208    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.691365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.690891    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.712414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.708767    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.796557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.796513    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:52:47.881564 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.881523    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:47.926934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.926894    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:47.927196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:47.927163    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:48.108422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.107759    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.226836 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.226726    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.247635 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.247583    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.486896 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.486795    5318 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Nov 03 20:52:48.589490 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.589433    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.590530 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.590504    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:52:48.747465 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.741448    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.747465 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.742180    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:52:48.764358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.755822    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.764358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.756578    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:52:48.934386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.903984    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:52:48.934386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.904739    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:52:48.934386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.904882    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:48.934386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.905780    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:52:48.948373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.935516    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:48.948688 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:48.948645    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:49.120421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.063438    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:49.120421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.063543    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.120421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.110607    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.120421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.114453    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:52:49.171362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.170321    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:49.171362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.170450    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.227566 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.227522    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.227945 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.227921    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:52:49.381417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.309867    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.457165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.457112    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.639418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.591446    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.891197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.891099    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.978380 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.976657    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:49.978380 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.978005    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:52:49.999268 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:49.999225    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.000279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.000247    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:50.023166 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.023120    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:50.109362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.102053    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:52:50.242421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.223958    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 20:52:50.269378 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.261501    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.340363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.336665    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:52:50.340363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.336737    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.429136 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.399806    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 20:52:50.580808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.580704    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.642362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.634498    5318 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.686531 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.686469    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:52:50.745369 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.724784    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:52:50.777516 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.777473    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:50.921866 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:50.908550    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:52:51.052388 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.052027    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:51.073201 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.065497    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:52:51.110824 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.093110    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:51.110824 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.093183    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:51.363561 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.363276    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:52:51.472496 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.472457    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:51.586894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.586844    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:51.648138 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.648054    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 20:52:51.653958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.653935    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 20:52:51.721553 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.720686    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 20:52:51.721553 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.721041    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 20:52:51.727490 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.727432    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:52:51.749879 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.749846    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 20:52:51.752829 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.752810    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 20:52:51.754369 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.753698    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 20:52:51.925082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.924979    5318 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:52:51.925082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:51.925061    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:52:53.374766 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:53.374727    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:53.375254 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:53.375131    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:53.376092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:53.376069    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 20:52:53.474737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:53.474706    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:53.680414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:53.680303    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:54.363376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:54.363326    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:54.571670 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:54.571579    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:54.806273 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:54.806240    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:55.427529 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:55.427497    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:55.631499 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:55.631468    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:55.843540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:55.843511    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:55.939888 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:52:55.939854    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:52:56.219588 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:56.219496    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:56.544081 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:56.543995    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:56.911897 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:56.911860    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:58.089620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.089579    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"]
Nov 03 20:52:58.093802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.093775    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140320    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140409    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140460    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140509    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140590    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pph6\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140622    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140714    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140738    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.140938 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.140778    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.188785 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.188752    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"]
Nov 03 20:52:58.241663 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241626    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.241663 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241665    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.241969 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241695    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.241969 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241777    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.241969 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241821    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.241969 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241858    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.241969 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241900    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.241974    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pph6\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.242006    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.242113    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.242173    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242478 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.242298    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242556 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.242534    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.242690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.242671    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.244112 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.244090    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.244364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.244328    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.299371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.299321    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.299965 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.299940    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pph6\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.411645 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.411562    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:58.830664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.830636    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"]
Nov 03 20:52:58.833191 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:52:58.833146    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773466d7_2d8c_4b3f_bd3e_c2f335ef0b35.slice/crio-e8f28885b4e72419e98b66449ae34ea35fee029aecee6d0ba2a2039904f13971 WatchSource:0}: Error finding container e8f28885b4e72419e98b66449ae34ea35fee029aecee6d0ba2a2039904f13971: Status 404 returned error can't find the container with id e8f28885b4e72419e98b66449ae34ea35fee029aecee6d0ba2a2039904f13971
Nov 03 20:52:58.835939 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.835903    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.836059 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.835966    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:58.836059 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:58.835997    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:59.203279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.203241    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" event={"ID":"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35","Type":"ContainerStarted","Data":"f8c4653a19e264337c57f327e791d7eebf17e6fcf6db3bfcc847c85d4c528a7c"}
Nov 03 20:52:59.203737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.203308    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" event={"ID":"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35","Type":"ContainerStarted","Data":"e8f28885b4e72419e98b66449ae34ea35fee029aecee6d0ba2a2039904f13971"}
Nov 03 20:52:59.323030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.322975    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podStartSLOduration=1.322959581 podStartE2EDuration="1.322959581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:59.319416119 +0000 UTC m=+1372.298972989" watchObservedRunningTime="2026-11-03 20:52:59.322959581 +0000 UTC m=+1372.302516468"
Nov 03 20:52:59.412669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.412636    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:52:59.413993 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.413955    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:52:59.414154 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.414004    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:52:59.416544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.416519    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:59.621130 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.621097    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:59.840238 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:52:59.840209    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:00.127279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:00.127248    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:00.412596 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:00.412526    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:53:00.412596 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:00.412572    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:53:01.186361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:01.183817    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:01.391982 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:01.391932    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:01.411878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:01.411851    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:53:01.412013 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:01.411905    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:53:01.636221 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:01.636179    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:01.919464 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:01.919388    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:02.200036 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:02.193942    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:02.405540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:02.405505    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:02.413519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:02.413484    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:53:02.413663 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:02.413561    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:53:03.412354 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:03.412311    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:53:03.412835 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:03.412373    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:53:03.536685 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:03.536647    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:03.771740 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:03.768206    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:04.003375 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.000905    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"]
Nov 03 20:53:04.009270 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.009247    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.047449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.046686    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"]
Nov 03 20:53:04.061291 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.061256    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.061861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.061833    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brc2c\" (UniqueName: \"kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c\") pod \"payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.080632 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.080605    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b6eaffa8\""
Nov 03 20:53:04.081004 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.080983    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b6eaffa8-dockercfg-j7b5t\""
Nov 03 20:53:04.164143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.164105    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brc2c\" (UniqueName: \"kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c\") pod \"payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.164433 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.164415    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.190360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.166150    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.223052 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.223017    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brc2c\" (UniqueName: \"kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c\") pod \"payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") " pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.345671 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.345644    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:04.414741 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.414394    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:53:04.414741 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.414449    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:53:04.745168 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:04.745085    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"]
Nov 03 20:53:05.268785 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:05.268750    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr" event={"ID":"7237dbc3-a76f-408a-a2ea-1422e417411a","Type":"ContainerStarted","Data":"de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117"}
Nov 03 20:53:05.268994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:05.268813    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr" event={"ID":"7237dbc3-a76f-408a-a2ea-1422e417411a","Type":"ContainerStarted","Data":"e5abb73aa890ae48b68ec118387f1c5b8ffcbf095fb7425d52ae511586a51bc0"}
Nov 03 20:53:05.268994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:05.268867    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:05.380071 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:05.380012    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr" podStartSLOduration=2.379382517 podStartE2EDuration="2.379382517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:05.378968406 +0000 UTC m=+1378.358525246" watchObservedRunningTime="2026-11-03 20:53:05.379382517 +0000 UTC m=+1378.358939359"
Nov 03 20:53:05.412723 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:05.412686    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 20:53:05.412873 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:05.412749    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 20:53:06.416552 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:06.416524    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:53:06.417022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:06.416997    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:53:06.417701 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:06.417680    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:53:06.537724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:06.537702    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:06.878462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:06.878424    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:07.101285 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:07.101254    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:07.445486 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:07.445455    5318 scope.go:122] "RemoveContainer" containerID="3bba59f376b82533024e5ed6ab561566bad0e7fe2e98913a1c2e9ae696f58e0d"
Nov 03 20:53:07.606115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:07.606078    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:08.737549 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:08.737521    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:08.942315 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:08.942277    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:11.603224 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.603194    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"]
Nov 03 20:53:11.608634 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.608610    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.702653    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.702887    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.702952    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.703003    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.703053    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.703100    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.703136    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.703260    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jl97\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.710355 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.703439    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.711713 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.711689    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"]
Nov 03 20:53:11.804247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804216    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jl97\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804302    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804356    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804395    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804419    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804450    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804494    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804522    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804555    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804917 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804827    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.804917 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.804878    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.805079 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.805057    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.805142 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.805101    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.805314 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.805289    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.806941 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.806915    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.807136 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.807120    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.862036 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.861976    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.862170 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.862140    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jl97\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:11.923315 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:11.923292    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:12.357406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.357377    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"]
Nov 03 20:53:12.358787 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:53:12.358763    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7b6894_4449_4178_ab06_faaf1df1351e.slice/crio-4d5f224addd421bb605bb050170d0871d6b03da9a8558b08c607df65cadbdd72 WatchSource:0}: Error finding container 4d5f224addd421bb605bb050170d0871d6b03da9a8558b08c607df65cadbdd72: Status 404 returned error can't find the container with id 4d5f224addd421bb605bb050170d0871d6b03da9a8558b08c607df65cadbdd72
Nov 03 20:53:12.361182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.361150    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:12.361266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.361210    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:12.361266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.361237    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:12.458386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.458045    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:12.534241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.534208    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"]
Nov 03 20:53:12.541356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.540731    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.619395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.617054    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"]
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622233    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622291    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622320    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622444    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622483    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wznk\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622561    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622591    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622662    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.623618 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.622755    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723593    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723626 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723629    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723671    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723732    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723792    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723825    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.723854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723852    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724109 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723957    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724109 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.723993    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wznk\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724216 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.724104    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724330 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.724303    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724432 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.724348    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.724547    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.724655 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.724631    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.725941 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.725920    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.726760 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.726734    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.781440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.781412    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.781625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.781603    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wznk\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:12.856499 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:12.856474    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:13.190542 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.190422    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"]
Nov 03 20:53:13.195648 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.195624    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.257319 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.257290    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"]
Nov 03 20:53:13.295360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.294627    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"]
Nov 03 20:53:13.320716 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.320597    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.320716 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.320684    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.320716 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.320728    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.329054 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.328047    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" event={"ID":"8e7b6894-4449-4178-ab06-faaf1df1351e","Type":"ContainerStarted","Data":"b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f"}
Nov 03 20:53:13.329054 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.328105    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" event={"ID":"8e7b6894-4449-4178-ab06-faaf1df1351e","Type":"ContainerStarted","Data":"4d5f224addd421bb605bb050170d0871d6b03da9a8558b08c607df65cadbdd72"}
Nov 03 20:53:13.329907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.329878    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" event={"ID":"e15d0a92-f1a6-4d7e-a803-d0ee269529f9","Type":"ContainerStarted","Data":"2ab3816071448616b4d0f4493c9c8d5dc9ac3f5c75c81716e166a4d2717efcd9"}
Nov 03 20:53:13.332837 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.332809    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.332934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.332854    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.332934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.332916    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.333057 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.332988    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.333057 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.333013    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckw9\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.333057 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.333049    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.333204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.333134    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.333204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.333192    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.333310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.333227    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434169    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434312 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434210    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434312 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434274    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434429 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434390    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434429 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434419    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckw9\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434513 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434455    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434640 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434614    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434682 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434643    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434694    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434759 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434739    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.434903 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.434882    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.435162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.435135    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.435296 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.435276    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.435554 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.435520    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.437501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.437477    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.437678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.437661    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.455770 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.455684    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podStartSLOduration=2.455671907 podStartE2EDuration="2.455671907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:13.453655952 +0000 UTC m=+1386.433212818" watchObservedRunningTime="2026-11-03 20:53:13.455671907 +0000 UTC m=+1386.435228748"
Nov 03 20:53:13.504359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.503724    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckw9\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.504359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.504109    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.515966 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.515934    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:13.650590 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.650560    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:13.857583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.857520    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:13.924450 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.924421    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:13.925143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.925115    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:13.925202 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.925175    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:13.942376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.939618    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"]
Nov 03 20:53:13.946218 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.946184    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.946324 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.946244    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:13.946324 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:13.946275    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:14.347569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.347427    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" event={"ID":"2f928666-acf3-4f06-bcad-1c4ae20c49c1","Type":"ContainerStarted","Data":"43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1"}
Nov 03 20:53:14.347569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.347493    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" event={"ID":"2f928666-acf3-4f06-bcad-1c4ae20c49c1","Type":"ContainerStarted","Data":"48b249065b9cbd255e5846040bac8f9c7867e2d15ec11ae9360741f0331a8821"}
Nov 03 20:53:14.362974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.362721    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" event={"ID":"e15d0a92-f1a6-4d7e-a803-d0ee269529f9","Type":"ContainerStarted","Data":"01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be"}
Nov 03 20:53:14.497411 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.493352    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podStartSLOduration=1.493311441 podStartE2EDuration="1.493311441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:14.464730919 +0000 UTC m=+1387.444287759" watchObservedRunningTime="2026-11-03 20:53:14.493311441 +0000 UTC m=+1387.472868282"
Nov 03 20:53:14.519408 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.517305    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:14.615364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.613758    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podStartSLOduration=2.61373884 podStartE2EDuration="2.61373884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:14.591891066 +0000 UTC m=+1387.571447907" watchObservedRunningTime="2026-11-03 20:53:14.61373884 +0000 UTC m=+1387.593295680"
Nov 03 20:53:14.857564 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.857511    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:14.858427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.858400    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:14.858583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.858452    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:14.928767 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.928734    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:14.929059 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:14.929019    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:15.523888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:15.523544    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:15.523888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:15.523605    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:15.868363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:15.861227    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:15.868363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:15.861291    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:15.926358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:15.925889    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:15.926358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:15.926002    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:16.319416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.318553    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:53:16.575423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.550870    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:16.575423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.550951    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:16.870586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.862771    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:16.870586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.862831    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:16.964959 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.964928    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:16.965170 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:16.965150    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:17.490393 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.490071    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:17.519379 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.516694    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:17.519379 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.516752    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:17.858814 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.858781    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:17.859077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.859043    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:17.933614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.933578    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:17.959057 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:17.958193    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:18.318421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.310896    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:18.551811 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.551325    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:18.551811 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.551418    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:18.857283 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.857241    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:18.857677 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.857561    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:18.923904 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.923822    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:18.924203 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:18.924153    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:19.534360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:19.531794    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:19.534360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:19.531861    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:19.857512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:19.857477    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:19.857703 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:19.857536    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:19.928362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:19.926040    5318 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:53:19.928362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:19.926102    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:53:20.516932 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:20.516759    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:20.516932 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:20.516832    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:20.625380 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:20.625175    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:20.857328 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:20.857295    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:53:20.857511 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:20.857379    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:53:20.862299 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:20.862272    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:21.216758 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:21.216669    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:21.218300 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:21.218279    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:21.227584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:21.227561    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:53:21.436382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:21.423063    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:21.521661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:21.519489    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:21.521661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:21.521159    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:22.230882 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.230824    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:22.419213 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.419184    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:22.420135 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.420108    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:22.421363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.421323    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:53:22.516777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.516697    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:53:22.516777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.516763    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:53:22.690207 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:22.690171    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:23.089017 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:23.088983    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:53:23.525985 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:23.525881    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:23.526994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:23.526971    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:23.528003 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:23.527979    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:53:25.834219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:25.834181    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:25.896518 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:25.896478    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:26.134385 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:53:26.124520    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:53:26.179478 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:26.179393    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:26.388804 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:26.388724    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:27.181359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.177435    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:27.453634 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.453543    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:27.651687 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.651655    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"]
Nov 03 20:53:27.704033 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.703963    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"]
Nov 03 20:53:27.704393 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.704377    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"]
Nov 03 20:53:27.708860 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.704304    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.720000 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.719065    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.784077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.784040    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.784375 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.784350    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.790726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.790696    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.790930 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.790907    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.791057 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.791039    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.791192 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.791174    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.791310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.791294    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.791586 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.791566    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxcd\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.791716 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.791700    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.839801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.839764    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"]
Nov 03 20:53:27.907395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.904771    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:27.907395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907173    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.907395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907254    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.907395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907295    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907405    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907451    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907481    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907517    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5hz\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907568    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907609    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907649    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907695    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.907744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907729    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.908240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907768    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.908240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907861    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.908240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.907897    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.908240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.908003    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:27.908240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.908046    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxcd\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.908240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.908089    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.908661 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.908636    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.919138 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.919111    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.942747 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.919520    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.942747 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.919819    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.942747 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.939912    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.942747 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.940208    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.947933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.947886    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.982743 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.982565    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:27.988452 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:27.988417    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxcd\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010134    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010192    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010218    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5hz\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010286    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010355    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010440    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010471    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010591    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.011418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.010771    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.013203 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.011615    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.013566 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.013535    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.014410 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.013748    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.014498 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.013974    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.014498 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.014273    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.014774 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.014757    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.021784 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.021756    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.070867 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.070830    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.072501 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.072476    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5hz\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.126601 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.126569    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:28.139076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.139051    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:28.207382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.206578    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:28.565477 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.565444    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"]
Nov 03 20:53:28.566381 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:53:28.566326    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c13e2a6_38b9_4429_aa83_c5b1e0ec60e0.slice/crio-1e117a50df1074efa90a97598d6c496554c666cca4eb2352f827bad5305bd721 WatchSource:0}: Error finding container 1e117a50df1074efa90a97598d6c496554c666cca4eb2352f827bad5305bd721: Status 404 returned error can't find the container with id 1e117a50df1074efa90a97598d6c496554c666cca4eb2352f827bad5305bd721
Nov 03 20:53:28.569066 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.569035    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:28.569144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.569096    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:28.569144 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.569122    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:28.622958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.622933    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"]
Nov 03 20:53:28.624292 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:53:28.624269    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8cfe95_a734_4615_aad1_910e01c0e9e1.slice/crio-312c4046549dfb4a12ff103166d629e68c49bc68bec38c9cc80362b9c5b916aa WatchSource:0}: Error finding container 312c4046549dfb4a12ff103166d629e68c49bc68bec38c9cc80362b9c5b916aa: Status 404 returned error can't find the container with id 312c4046549dfb4a12ff103166d629e68c49bc68bec38c9cc80362b9c5b916aa
Nov 03 20:53:28.626931 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.626903    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:28.627030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.626962    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:28.627030 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.626990    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:28.909507 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.909471    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" event={"ID":"1b8cfe95-a734-4615-aad1-910e01c0e9e1","Type":"ContainerStarted","Data":"510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14"}
Nov 03 20:53:28.909703 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.909536    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" event={"ID":"1b8cfe95-a734-4615-aad1-910e01c0e9e1","Type":"ContainerStarted","Data":"312c4046549dfb4a12ff103166d629e68c49bc68bec38c9cc80362b9c5b916aa"}
Nov 03 20:53:28.913430 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.913403    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" event={"ID":"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0","Type":"ContainerStarted","Data":"133d88ff99f3947180f10cf8ae34ee6d58f510eebc77a8b242f49edf1adc155c"}
Nov 03 20:53:28.913635 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:28.913615    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" event={"ID":"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0","Type":"ContainerStarted","Data":"1e117a50df1074efa90a97598d6c496554c666cca4eb2352f827bad5305bd721"}
Nov 03 20:53:29.035224 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.035162    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podStartSLOduration=2.035144158 podStartE2EDuration="2.035144158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:29.032663461 +0000 UTC m=+1402.012220302" watchObservedRunningTime="2026-11-03 20:53:29.035144158 +0000 UTC m=+1402.014701000"
Nov 03 20:53:29.139619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.139579    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:29.140939 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.140911    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:29.141082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.140977    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:29.154881 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.154789    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podStartSLOduration=2.154769269 podStartE2EDuration="2.154769269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:29.153702237 +0000 UTC m=+1402.133259102" watchObservedRunningTime="2026-11-03 20:53:29.154769269 +0000 UTC m=+1402.134326111"
Nov 03 20:53:29.207807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.207712    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:29.209472 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.209445    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:29.209638 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.209488    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:29.864062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:29.864011    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:30.111857 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:30.108176    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:30.142851 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:30.141828    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:30.142851 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:30.141920    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:30.233364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:30.227773    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:30.233364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:30.227835    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:31.139536 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:31.139494    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:31.139718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:31.139555    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:31.220524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:31.220429    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:31.220524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:31.220497    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:32.140095 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:32.140055    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:32.140583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:32.140119    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:32.211890 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:32.211856    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:32.212197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:32.212154    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:33.139405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:33.139364    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:33.139699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:33.139425    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:33.216491 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:33.216448    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:33.216973 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:33.216514    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:34.139636 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:34.139606    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:34.139889 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:34.139861    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:34.217476 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:34.217440    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:34.218039 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:34.218008    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:35.140194 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:35.140156    5318 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:53:35.140407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:35.140223    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:53:35.207247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:35.207212    5318 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:53:35.207393 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:35.207273    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:53:35.488898 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:35.488789    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:36.315143 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:36.315111    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:36.315940 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:36.315916    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:36.316813 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:36.316794    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 20:53:36.377891 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:36.377864    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:36.378190 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:36.378171    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:36.379126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:36.379109    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:53:38.832588 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:38.832557    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:40.391303 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:40.391256    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:43.425871 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.425824    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"]
Nov 03 20:53:43.433904 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.433877    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.510732 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.510695    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.510914 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.510819    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.510914 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.510852    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.511022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.510942    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.511022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.510986    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.511098 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.511028    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.511145 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.511127    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.511204 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.511188    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.511266 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.511249    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rk8t\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.540512 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.540475    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"]
Nov 03 20:53:43.612623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612585    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612635    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612682    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612723    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612765    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612965 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612815    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612965 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612889    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.612965 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.612923    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rk8t\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.613089 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.613064    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.613149 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.613110    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.613313 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.613294    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.613625 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.613600    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.613711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.613600    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.613808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.613783    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.615462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.615435    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.615654 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.615638    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.669947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.669918    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.670054 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.669989    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rk8t\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:43.748138 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:43.748085    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:44.173538 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.173508    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"]
Nov 03 20:53:44.219773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.219734    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:44.219993 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.219972    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:44.220136 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.220119    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:44.672166 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.672135    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"]
Nov 03 20:53:44.677413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.677392    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.726956 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.726913    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b6eaffa8-serving-cert\""
Nov 03 20:53:44.730943 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.730918    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"]
Nov 03 20:53:44.844094 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.844054    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl9j8\" (UniqueName: \"kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.844290 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.844147    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.844383 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.844364    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.844520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.844504    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.911729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.911691    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:44.946327 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.946242    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.946327 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.946292    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl9j8\" (UniqueName: \"kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.946597 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.946389    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.946689 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.946668    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.947040 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.947022    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.949210 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.949164    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:44.949329 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:44.949312    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:45.004128 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.004090    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl9j8\" (UniqueName: \"kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8\") pod \"maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:45.029022 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.028988    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" event={"ID":"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c","Type":"ContainerStarted","Data":"c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72"}
Nov 03 20:53:45.029201 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.029047    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" event={"ID":"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c","Type":"ContainerStarted","Data":"f68f3a7e2683528cffabb49bcfe04a27da4f066cdbf62798167f3b3f3a8e3ab7"}
Nov 03 20:53:45.111393 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.111304    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:45.152893 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.152834    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podStartSLOduration=2.152815896 podStartE2EDuration="2.152815896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:45.152085768 +0000 UTC m=+1418.131642621" watchObservedRunningTime="2026-11-03 20:53:45.152815896 +0000 UTC m=+1418.132372736"
Nov 03 20:53:45.292423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.292282    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:45.700004 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.699957    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"]
Nov 03 20:53:45.700714 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:53:45.700684    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f913fa0_4e35_4597_a714_0ef81fe5f7b4.slice/crio-26fc956da6fbd686c53f4531c485e9dbe4813232c390f89f61e0a715d2d1001d WatchSource:0}: Error finding container 26fc956da6fbd686c53f4531c485e9dbe4813232c390f89f61e0a715d2d1001d: Status 404 returned error can't find the container with id 26fc956da6fbd686c53f4531c485e9dbe4813232c390f89f61e0a715d2d1001d
Nov 03 20:53:45.702223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.702204    5318 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:53:45.749178 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.749153    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:45.750419 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.750392    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:53:45.750524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:45.750445    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:53:46.037607 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.037525    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6" event={"ID":"5f913fa0-4e35-4597-a714-0ef81fe5f7b4","Type":"ContainerStarted","Data":"26fc956da6fbd686c53f4531c485e9dbe4813232c390f89f61e0a715d2d1001d"}
Nov 03 20:53:46.117566 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.117534    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"]
Nov 03 20:53:46.123008 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.122989    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.185357 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.173827    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\""
Nov 03 20:53:46.223913 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.223880    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"]
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.289766    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.289840    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.289878    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.289998    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.290058    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.290120    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.290185    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpcs\" (UniqueName: \"kubernetes.io/projected/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-kube-api-access-5kpcs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.290220    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.290282    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.392356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.392188    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.392356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.392273    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.392356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.392352    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.395620    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.396412    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.392422    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpcs\" (UniqueName: \"kubernetes.io/projected/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-kube-api-access-5kpcs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.396555    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.396623    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.396687    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.396761    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.396802    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.401755    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.402666    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.403200    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.406381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.405159    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.414421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.413843    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.453827 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.453806    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.457791 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.457769    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpcs\" (UniqueName: \"kubernetes.io/projected/e69b42ce-7354-47e8-ac8f-22aeb8b0a70a-kube-api-access-5kpcs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr\" (UID: \"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.747361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.736887    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:46.749084 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.748646    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:53:46.749084 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:46.748702    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:53:47.051415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.050616    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6" event={"ID":"5f913fa0-4e35-4597-a714-0ef81fe5f7b4","Type":"ContainerStarted","Data":"7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467"}
Nov 03 20:53:47.051415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.050783    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:47.250712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.239633    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6" podStartSLOduration=2.150161509 podStartE2EDuration="3.239614466s" totalImagesPullingTime="1.089452957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:47.207624742 +0000 UTC m=+1420.187181583" watchObservedRunningTime="2026-11-03 20:53:47.239614466 +0000 UTC m=+1420.219171308"
Nov 03 20:53:47.250712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.241517    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"]
Nov 03 20:53:47.250712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.243125    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:47.250712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.243266    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:47.250712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.243397    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:53:47.748805 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.748773    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:53:47.749286 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:47.748843    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:53:48.060162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.060064    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" event={"ID":"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a","Type":"ContainerStarted","Data":"e62b95ba93ed9d07646e7331408db7bcab058e0e7a5d5209143b6fce72387280"}
Nov 03 20:53:48.060162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.060131    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" event={"ID":"e69b42ce-7354-47e8-ac8f-22aeb8b0a70a","Type":"ContainerStarted","Data":"924ff01684278f5b599b28aee747d713779b798d5c0f7162d157d6f89f4e9f25"}
Nov 03 20:53:48.185361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.182602    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podStartSLOduration=2.182583264 podStartE2EDuration="2.182583264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:48.176061575 +0000 UTC m=+1421.155618415" watchObservedRunningTime="2026-11-03 20:53:48.182583264 +0000 UTC m=+1421.162140105"
Nov 03 20:53:48.740071 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.740037    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:48.741407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.741379    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:48.741499 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.741429    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:48.749160 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.749138    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:53:48.749524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:48.749174    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:53:49.109360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.109307    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"]
Nov 03 20:53:49.131359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.130428    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.165226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.165198    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"]
Nov 03 20:53:49.208131 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.208091    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.208431 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.208405    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.208620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.208602    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.209017 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.208994    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxwb\" (UniqueName: \"kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.238363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.224206    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15564eb1-serving-cert\""
Nov 03 20:53:49.338363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.337535    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.338363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.337582    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.338363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.337616    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.338363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.337758    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxwb\" (UniqueName: \"kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.340396 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:53:49.339541    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-serving-cert: secret "maas-api-e2e-ait-15564eb1-serving-cert" not found
Nov 03 20:53:49.340396 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:53:49.339612    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls podName:1e98d437-632e-4819-a697-b07c7423bd74 nodeName:}" failed. No retries permitted until 2026-11-03 20:53:49.83959204 +0000 UTC m=+1422.819148866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls") pod "maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" (UID: "1e98d437-632e-4819-a697-b07c7423bd74") : secret "maas-api-e2e-ait-15564eb1-serving-cert" not found
Nov 03 20:53:49.340396 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.340005    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.341125 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.341098    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.358614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.358589    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:49.455407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.455034    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxwb\" (UniqueName: \"kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.623440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.622199    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:49.748447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.745855    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:49.748447 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.745916    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:49.748727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.748699    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:53:49.748934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.748908    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:53:49.850531 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.850492    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:49.853453 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:49.853432    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\") pod \"maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:50.048825 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:50.048749    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:50.553961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:50.553930    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"]
Nov 03 20:53:50.555247 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:53:50.555185    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e98d437_632e_4819_a697_b07c7423bd74.slice/crio-2d1bb02a7028d7c39312c63d7ded07d8ae571c58f9fc2a328af84774ba0ffc22 WatchSource:0}: Error finding container 2d1bb02a7028d7c39312c63d7ded07d8ae571c58f9fc2a328af84774ba0ffc22: Status 404 returned error can't find the container with id 2d1bb02a7028d7c39312c63d7ded07d8ae571c58f9fc2a328af84774ba0ffc22
Nov 03 20:53:50.740308 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:50.740272    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:50.740495 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:50.740350    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:50.748665 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:50.748641    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:53:50.748769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:50.748693    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:53:51.159737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:51.159702    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" event={"ID":"1e98d437-632e-4819-a697-b07c7423bd74","Type":"ContainerStarted","Data":"2d1bb02a7028d7c39312c63d7ded07d8ae571c58f9fc2a328af84774ba0ffc22"}
Nov 03 20:53:51.744357 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:51.743635    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:51.744357 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:51.743684    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:52.215304 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.205825    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" event={"ID":"1e98d437-632e-4819-a697-b07c7423bd74","Type":"ContainerStarted","Data":"4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43"}
Nov 03 20:53:52.215304 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.215239    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:52.228552 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.228528    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:52.229479 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.229452    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:52.232715 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.232693    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:53:52.464165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.464106    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" podStartSLOduration=2.581442601 podStartE2EDuration="3.464087253s" totalImagesPullingTime="882.644652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:52.330762938 +0000 UTC m=+1425.310319790" watchObservedRunningTime="2026-11-03 20:53:52.464087253 +0000 UTC m=+1425.443644093"
Nov 03 20:53:52.739788 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.739757    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:52.739964 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:52.739829    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:53.740068 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:53.740032    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:53.740598 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:53.740085    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:54.087129 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:54.087102    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:53:54.390878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:54.390786    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:54.740685 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:54.740607    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:53:54.740685 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:54.740670    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:53:55.935849 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:53:55.935744    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:53:56.562211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:56.562185    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:56.562584 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:56.562564    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:56.563175 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:56.563158    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr"
Nov 03 20:53:58.221467 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:58.221429    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:53:59.598488 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.598431    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:53:59.759860 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.759810    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:53:59.779069 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.779030    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:53:59.824103 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.824065    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"]
Nov 03 20:53:59.836543 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.836517    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:53:59.885914 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.885883    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"]
Nov 03 20:53:59.966470 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:53:59.966435    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f9337cba-serving-cert\""
Nov 03 20:54:00.052362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.026968    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:54:00.052362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.037719    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvbz\" (UniqueName: \"kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.052362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.037883    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.052362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.038083    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.052362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.038114    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.065381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.053075    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.065381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.058166    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.112546 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.082634    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:00.112933 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.077170    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.139458    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.139632    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.139662    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.139786    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvbz\" (UniqueName: \"kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:00.140374    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-serving-cert: secret "maas-api-e2e-ait-f9337cba-serving-cert" not found
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:00.140457    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls podName:51c8757b-41f2-475b-822c-411047d525ac nodeName:}" failed. No retries permitted until 2026-11-03 20:54:00.640434696 +0000 UTC m=+1433.619991539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls") pod "maas-api-e2e-ait-f9337cba-86b488c589-n827v" (UID: "51c8757b-41f2-475b-822c-411047d525ac") : secret "maas-api-e2e-ait-f9337cba-serving-cert" not found
Nov 03 20:54:00.143449 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.141350    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.209360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.206058    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.232876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.230423    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.240368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.237120    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvbz\" (UniqueName: \"kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.297371 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.284424    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:00.451722 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.428999    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:00.644359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.644199    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:00.708422 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:00.705736    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:00.708422 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:00.706164    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-serving-cert: secret "maas-api-e2e-ait-f9337cba-serving-cert" not found
Nov 03 20:54:00.708422 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:00.706234    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls podName:51c8757b-41f2-475b-822c-411047d525ac nodeName:}" failed. No retries permitted until 2026-11-03 20:54:01.706213902 +0000 UTC m=+1434.685770745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls") pod "maas-api-e2e-ait-f9337cba-86b488c589-n827v" (UID: "51c8757b-41f2-475b-822c-411047d525ac") : secret "maas-api-e2e-ait-f9337cba-serving-cert" not found
Nov 03 20:54:01.292865 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.292819    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"]
Nov 03 20:54:01.298425 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.298395    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.356667 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.356635    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"]
Nov 03 20:54:01.425446 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425411    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.425652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425460    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzhg\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.425652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425560    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.425652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425629    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.425832 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425749    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.425895 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425835    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.425951 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.425894    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.426067 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.426045    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.426114 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.426093    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.527913 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.527877    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.527929    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528075    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528232 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528104    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzhg\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528232 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528135    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528186    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528643    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528767    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528827    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.528876 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528839    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.529032 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.528903    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.529126 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.529098    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.529246 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.529148    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.529947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.529921    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.531293 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.531263    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.543260 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.543205    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.586329 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.586286    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.586526 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.586418    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzhg\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-t86gg\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.614919 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.614884    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:01.731953 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:01.731916    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:01.732350 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:01.732043    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-serving-cert: secret "maas-api-e2e-ait-f9337cba-serving-cert" not found
Nov 03 20:54:01.732350 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:01.732107    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls podName:51c8757b-41f2-475b-822c-411047d525ac nodeName:}" failed. No retries permitted until 2026-11-03 20:54:03.732088413 +0000 UTC m=+1436.711645234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls") pod "maas-api-e2e-ait-f9337cba-86b488c589-n827v" (UID: "51c8757b-41f2-475b-822c-411047d525ac") : secret "maas-api-e2e-ait-f9337cba-serving-cert" not found
Nov 03 20:54:02.036162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.036136    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"]
Nov 03 20:54:02.037881 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:54:02.037848    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245e503f_5797_4d03_be60_5b8c6ee8f741.slice/crio-6f21a99138cad5741b4ebbc3c7a74494d17f0f619fadfcedb1cd66b5738969cf WatchSource:0}: Error finding container 6f21a99138cad5741b4ebbc3c7a74494d17f0f619fadfcedb1cd66b5738969cf: Status 404 returned error can't find the container with id 6f21a99138cad5741b4ebbc3c7a74494d17f0f619fadfcedb1cd66b5738969cf
Nov 03 20:54:02.040466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.040440    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:02.040528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.040496    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:02.040528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.040525    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:02.278386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.277537    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:02.351579 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.351544    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" event={"ID":"245e503f-5797-4d03-be60-5b8c6ee8f741","Type":"ContainerStarted","Data":"3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6"}
Nov 03 20:54:02.351772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.351597    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" event={"ID":"245e503f-5797-4d03-be60-5b8c6ee8f741","Type":"ContainerStarted","Data":"6f21a99138cad5741b4ebbc3c7a74494d17f0f619fadfcedb1cd66b5738969cf"}
Nov 03 20:54:02.468955 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.468892    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podStartSLOduration=1.468877207 podStartE2EDuration="1.468877207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:02.465370377 +0000 UTC m=+1435.444927217" watchObservedRunningTime="2026-11-03 20:54:02.468877207 +0000 UTC m=+1435.448434047"
Nov 03 20:54:02.618609 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:02.618467    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:03.620809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:03.618842    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:03.620809 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:03.618914    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:03.766386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:03.766217    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:03.784191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:03.784110    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"maas-api-e2e-ait-f9337cba-86b488c589-n827v\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:03.794577 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:03.794498    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:04.226967 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:04.226935    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"]
Nov 03 20:54:04.228370 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:54:04.228332    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c8757b_41f2_475b_822c_411047d525ac.slice/crio-8e94b20111f8cc2065d691a5d40f5b7f037a0fe78089ce2b89d1d72826b09e81 WatchSource:0}: Error finding container 8e94b20111f8cc2065d691a5d40f5b7f037a0fe78089ce2b89d1d72826b09e81: Status 404 returned error can't find the container with id 8e94b20111f8cc2065d691a5d40f5b7f037a0fe78089ce2b89d1d72826b09e81
Nov 03 20:54:04.378532 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:04.378498    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v" event={"ID":"51c8757b-41f2-475b-822c-411047d525ac","Type":"ContainerStarted","Data":"8e94b20111f8cc2065d691a5d40f5b7f037a0fe78089ce2b89d1d72826b09e81"}
Nov 03 20:54:04.615756 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:04.615720    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:04.615947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:04.615783    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:05.402226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.402183    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v" event={"ID":"51c8757b-41f2-475b-822c-411047d525ac","Type":"ContainerStarted","Data":"d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c"}
Nov 03 20:54:05.402802 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.402377    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:05.515853 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.515793    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v" podStartSLOduration=5.692887264 podStartE2EDuration="6.51577755s" totalImagesPullingTime="822.890286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:05.5122964 +0000 UTC m=+1438.491853239" watchObservedRunningTime="2026-11-03 20:54:05.51577755 +0000 UTC m=+1438.495334390"
Nov 03 20:54:05.615760 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.615724    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:05.615920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.615793    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:05.694413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.693919    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:05.907381 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:05.907323    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:06.384358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:06.383744    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:06.616183 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:06.616144    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:06.616554 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:06.616207    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:07.616006 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:07.615972    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:07.616187 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:07.616046    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:08.616230 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:08.616194    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:08.616664 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:08.616265    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:09.615263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:09.615219    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:54:09.615474 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:09.615274    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:54:10.355358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:10.354383    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:11.070119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:11.070088    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:11.070873 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:11.070846    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:11.071594 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:11.071575    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 20:54:11.421588 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:11.421562    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:54:12.336974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:12.336941    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:14.922508 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:14.922465    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:15.922606 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:15.922527    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:16.137948 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:16.137913    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:18.865627 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:18.865592    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:19.465162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:19.465133    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:20.971223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:20.971192    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:23.917601 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:23.917510    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:25.925721 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:25.925688    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:25.933564 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:25.933532    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:54:28.662724 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:28.662681    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:33.164623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:33.164589    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:33.299109 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:33.299070    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:54:37.597550 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:37.597512    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:38.456557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:38.456522    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:38.731524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:38.731408    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:43.099629 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:43.099584    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:46.663735 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.663702    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"]
Nov 03 20:54:46.669289 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.669269    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.789715 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.789674    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"]
Nov 03 20:54:46.819769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.819729    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.819769 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.819766    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820010 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.819788    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsg7\" (UniqueName: \"kubernetes.io/projected/30f93857-ee14-46af-a80e-7d9e242e2b73-kube-api-access-jnsg7\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820010 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.819883    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820121 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.820060    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820193 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.820173    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f93857-ee14-46af-a80e-7d9e242e2b73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.820242    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.820314    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.820455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.820435    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921404 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921324    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921434    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f93857-ee14-46af-a80e-7d9e242e2b73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921523    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921558    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921642 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921618    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921843 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921802    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.921843 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921834    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921858    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsg7\" (UniqueName: \"kubernetes.io/projected/30f93857-ee14-46af-a80e-7d9e242e2b73-kube-api-access-jnsg7\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922058 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.921893    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922171 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.922137    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f93857-ee14-46af-a80e-7d9e242e2b73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.922362    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.922386    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922580 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.922561    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.922619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.922598    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.924136 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.924116    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.924569 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.924550    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.979165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.979140    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f93857-ee14-46af-a80e-7d9e242e2b73-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.979274 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.979256    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsg7\" (UniqueName: \"kubernetes.io/projected/30f93857-ee14-46af-a80e-7d9e242e2b73-kube-api-access-jnsg7\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt\" (UID: \"30f93857-ee14-46af-a80e-7d9e242e2b73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:46.983782 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:46.983759    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:47.409695 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.409667    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"]
Nov 03 20:54:47.411257 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:54:47.411225    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f93857_ee14_46af_a80e_7d9e242e2b73.slice/crio-ea1df3673f52607e0e7ee51e66e9ab7608f8432cbaffaa43caf93c0eca1a11bb WatchSource:0}: Error finding container ea1df3673f52607e0e7ee51e66e9ab7608f8432cbaffaa43caf93c0eca1a11bb: Status 404 returned error can't find the container with id ea1df3673f52607e0e7ee51e66e9ab7608f8432cbaffaa43caf93c0eca1a11bb
Nov 03 20:54:47.413801 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.413766    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.413860 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.413831    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.413895 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.413881    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:54:47.489050 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.489005    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:47.736180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.736082    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" event={"ID":"30f93857-ee14-46af-a80e-7d9e242e2b73","Type":"ContainerStarted","Data":"173b08eb3ae29dd1df16e62cf72209dccbb48349628edd3ea1f953590a4f95fd"}
Nov 03 20:54:47.736180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.736142    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" event={"ID":"30f93857-ee14-46af-a80e-7d9e242e2b73","Type":"ContainerStarted","Data":"ea1df3673f52607e0e7ee51e66e9ab7608f8432cbaffaa43caf93c0eca1a11bb"}
Nov 03 20:54:47.856180 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.856129    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podStartSLOduration=1.856110755 podStartE2EDuration="1.856110755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:47.852245706 +0000 UTC m=+1480.831802546" watchObservedRunningTime="2026-11-03 20:54:47.856110755 +0000 UTC m=+1480.835667634"
Nov 03 20:54:47.984249 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:47.984218    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:48.990364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:48.988422    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:48.990364 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:48.988503    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:49.987737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:49.987701    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:49.989158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:49.989122    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:50.985358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:50.984628    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:50.985358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:50.984689    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:51.984725 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:51.984688    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:51.985011 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:51.984983    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:52.987243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:52.987204    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:52.987880 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:52.987847    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:53.992946 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:53.984653    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:53.992946 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:53.984715    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:54.286138 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:54.286068    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:54.985092 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:54.985055    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 20:54:54.985257 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:54.985114    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 20:54:55.955589 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:54:55.955534    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:54:56.201846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:56.201805    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:56.202215 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:56.202198    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:56.203068 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:56.203044    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt"
Nov 03 20:54:59.507757 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:54:59.507702    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:03.545948 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:03.545912    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"]
Nov 03 20:55:04.064165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:04.064122    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:04.270958 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:04.270921    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:04.975360 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:04.972930    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:07.503438 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:07.503410    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:55:07.511797 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:07.511775    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 20:55:08.546862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.546795    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr" podUID="7237dbc3-a76f-408a-a2ea-1422e417411a" containerName="payload-processing" containerID="cri-o://de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117" gracePeriod=25
Nov 03 20:55:08.823329 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.823305    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:55:08.898269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.898241    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume\") pod \"7237dbc3-a76f-408a-a2ea-1422e417411a\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") "
Nov 03 20:55:08.898399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.898371    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c\" (UniqueName: \"kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c\") pod \"7237dbc3-a76f-408a-a2ea-1422e417411a\" (UID: \"7237dbc3-a76f-408a-a2ea-1422e417411a\") "
Nov 03 20:55:08.898616 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.898594    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume" pod "7237dbc3-a76f-408a-a2ea-1422e417411a" (UID: "7237dbc3-a76f-408a-a2ea-1422e417411a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:08.898840 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.898827    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7237dbc3-a76f-408a-a2ea-1422e417411a-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:08.900269 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.900252    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c" pod "7237dbc3-a76f-408a-a2ea-1422e417411a" (UID: "7237dbc3-a76f-408a-a2ea-1422e417411a"). InnerVolumeSpecName "kube-api-access-brc2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:08.929418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.929394    5318 generic.go:356] "Generic (PLEG): container finished" podID="7237dbc3-a76f-408a-a2ea-1422e417411a" containerID="de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117" exitCode=0
Nov 03 20:55:08.929528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.929441    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"
Nov 03 20:55:08.929528 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.929481    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr" event={"ID":"7237dbc3-a76f-408a-a2ea-1422e417411a","Type":"ContainerDied","Data":"de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117"}
Nov 03 20:55:08.929612 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.929531    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr" event={"ID":"7237dbc3-a76f-408a-a2ea-1422e417411a","Type":"ContainerDied","Data":"e5abb73aa890ae48b68ec118387f1c5b8ffcbf095fb7425d52ae511586a51bc0"}
Nov 03 20:55:08.929612 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.929559    5318 scope.go:122] "RemoveContainer" containerID="de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117"
Nov 03 20:55:08.942407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.942385    5318 scope.go:122] "RemoveContainer" containerID="de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117"
Nov 03 20:55:08.942740 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:55:08.942712    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117\": container with ID starting with de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117 not found: ID does not exist" containerID="de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117"
Nov 03 20:55:08.942835 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:08.942749    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117"} err="failed to get container status \"de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117\": rpc error: code = NotFound desc = could not find container \"de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117\": container with ID starting with de2af42bd165734a558f70158acf288912a972d18f059529334308981628a117 not found: ID does not exist"
Nov 03 20:55:09.000727 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:09.000697    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brc2c\" (UniqueName: \"kubernetes.io/projected/7237dbc3-a76f-408a-a2ea-1422e417411a-kube-api-access-brc2c\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:09.094374 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:09.094331    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"]
Nov 03 20:55:09.098505 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:09.098476    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b6eaffa8-758c7d9f75-8gwlr"]
Nov 03 20:55:09.257575 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:09.257538    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7237dbc3-a76f-408a-a2ea-1422e417411a" path="/var/lib/kubelet/pods/7237dbc3-a76f-408a-a2ea-1422e417411a/volumes"
Nov 03 20:55:10.353844 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:10.353812    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:13.049101 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:13.049064    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:13.252233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:13.252204    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:15.795543 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:15.795503    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:20.488071 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:20.488028    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:21.303019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:21.302968    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:22.356927 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:22.356848    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:22.559129 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:22.559094    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:25.147966 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:25.147930    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:25.934638 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:55:25.934601    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:55:27.488477 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:27.488436    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:55:29.947628 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:29.947587    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:33.252475 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:33.252438    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.136001 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:34.135970    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.348864 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:34.348831    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:34.769321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:34.769272    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:35.060853 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:35.060726    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:39.545035 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:39.545001    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:40.962796 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:40.962765    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:41.708983 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:41.708955    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:43.307244 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.307204    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"]
Nov 03 20:55:43.307763 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.307390    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6" podUID="5f913fa0-4e35-4597-a714-0ef81fe5f7b4" containerName="maas-api" containerID="cri-o://7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467" gracePeriod=30
Nov 03 20:55:43.466561 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.466519    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:43.632790 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.632759    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:55:43.836919 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.836871    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8\" (UniqueName: \"kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8\") pod \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") "
Nov 03 20:55:43.837118 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.836985    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls\") pod \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") "
Nov 03 20:55:43.837118 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.837046    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca\") pod \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") "
Nov 03 20:55:43.837280 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.837257    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs\") pod \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\" (UID: \"5f913fa0-4e35-4597-a714-0ef81fe5f7b4\") "
Nov 03 20:55:43.838414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.838384    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca" pod "5f913fa0-4e35-4597-a714-0ef81fe5f7b4" (UID: "5f913fa0-4e35-4597-a714-0ef81fe5f7b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:43.842146 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.842104    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls" pod "5f913fa0-4e35-4597-a714-0ef81fe5f7b4" (UID: "5f913fa0-4e35-4597-a714-0ef81fe5f7b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:43.842815 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.842781    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs" pod "5f913fa0-4e35-4597-a714-0ef81fe5f7b4" (UID: "5f913fa0-4e35-4597-a714-0ef81fe5f7b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:43.844253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.844218    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8" pod "5f913fa0-4e35-4597-a714-0ef81fe5f7b4" (UID: "5f913fa0-4e35-4597-a714-0ef81fe5f7b4"). InnerVolumeSpecName "kube-api-access-bl9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:43.940073 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.940033    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl9j8\" (UniqueName: \"kubernetes.io/projected/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-kube-api-access-bl9j8\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:43.940073 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.940073    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:43.940286 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.940086    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:43.940286 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:43.940099    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f913fa0-4e35-4597-a714-0ef81fe5f7b4-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:44.182182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.182100    5318 generic.go:356] "Generic (PLEG): container finished" podID="5f913fa0-4e35-4597-a714-0ef81fe5f7b4" containerID="7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467" exitCode=0
Nov 03 20:55:44.182182 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.182164    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"
Nov 03 20:55:44.182442 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.182205    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6" event={"ID":"5f913fa0-4e35-4597-a714-0ef81fe5f7b4","Type":"ContainerDied","Data":"7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467"}
Nov 03 20:55:44.182442 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.182285    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6" event={"ID":"5f913fa0-4e35-4597-a714-0ef81fe5f7b4","Type":"ContainerDied","Data":"26fc956da6fbd686c53f4531c485e9dbe4813232c390f89f61e0a715d2d1001d"}
Nov 03 20:55:44.182442 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.182315    5318 scope.go:122] "RemoveContainer" containerID="7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467"
Nov 03 20:55:44.196321 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.196303    5318 scope.go:122] "RemoveContainer" containerID="7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467"
Nov 03 20:55:44.196661 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:55:44.196641    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467\": container with ID starting with 7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467 not found: ID does not exist" containerID="7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467"
Nov 03 20:55:44.196750 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.196671    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467"} err="failed to get container status \"7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467\": rpc error: code = NotFound desc = could not find container \"7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467\": container with ID starting with 7399792e992005b802a2c46c6ce775a2cb379f1ff5524e38b7ae48fd1e910467 not found: ID does not exist"
Nov 03 20:55:44.336559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.336530    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"]
Nov 03 20:55:44.339898 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.339873    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b6eaffa8-b5d67f65-4xjp6"]
Nov 03 20:55:44.400250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.400221    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"]
Nov 03 20:55:44.400460 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.400429    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" podUID="1e98d437-632e-4819-a697-b07c7423bd74" containerName="maas-api" containerID="cri-o://4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43" gracePeriod=30
Nov 03 20:55:44.557004 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.556970    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:44.801284 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.800837    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:55:44.916363 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.915867    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:44.992635 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.992600    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs\") pod \"1e98d437-632e-4819-a697-b07c7423bd74\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") "
Nov 03 20:55:44.992819 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.992648    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\") pod \"1e98d437-632e-4819-a697-b07c7423bd74\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") "
Nov 03 20:55:44.992819 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.992717    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca\") pod \"1e98d437-632e-4819-a697-b07c7423bd74\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") "
Nov 03 20:55:44.992929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.992902    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb\" (UniqueName: \"kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb\") pod \"1e98d437-632e-4819-a697-b07c7423bd74\" (UID: \"1e98d437-632e-4819-a697-b07c7423bd74\") "
Nov 03 20:55:44.995983 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.995952    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca" pod "1e98d437-632e-4819-a697-b07c7423bd74" (UID: "1e98d437-632e-4819-a697-b07c7423bd74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:44.998382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:44.998314    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs" pod "1e98d437-632e-4819-a697-b07c7423bd74" (UID: "1e98d437-632e-4819-a697-b07c7423bd74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:45.000256 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.000229    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb" pod "1e98d437-632e-4819-a697-b07c7423bd74" (UID: "1e98d437-632e-4819-a697-b07c7423bd74"). InnerVolumeSpecName "kube-api-access-xpxwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:45.019830 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.019117    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls" pod "1e98d437-632e-4819-a697-b07c7423bd74" (UID: "1e98d437-632e-4819-a697-b07c7423bd74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:45.095480 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.095441    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:45.095480 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.095470    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e98d437-632e-4819-a697-b07c7423bd74-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:45.095480 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.095480    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e98d437-632e-4819-a697-b07c7423bd74-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:45.095480 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.095489    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxwb\" (UniqueName: \"kubernetes.io/projected/1e98d437-632e-4819-a697-b07c7423bd74-kube-api-access-xpxwb\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:45.181602 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.181542    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:45.194276 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.194246    5318 generic.go:356] "Generic (PLEG): container finished" podID="1e98d437-632e-4819-a697-b07c7423bd74" containerID="4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43" exitCode=0
Nov 03 20:55:45.194448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.194311    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"
Nov 03 20:55:45.194448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.194317    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" event={"ID":"1e98d437-632e-4819-a697-b07c7423bd74","Type":"ContainerDied","Data":"4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43"}
Nov 03 20:55:45.194448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.194401    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc" event={"ID":"1e98d437-632e-4819-a697-b07c7423bd74","Type":"ContainerDied","Data":"2d1bb02a7028d7c39312c63d7ded07d8ae571c58f9fc2a328af84774ba0ffc22"}
Nov 03 20:55:45.194448 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.194425    5318 scope.go:122] "RemoveContainer" containerID="4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43"
Nov 03 20:55:45.207149 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.207131    5318 scope.go:122] "RemoveContainer" containerID="4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43"
Nov 03 20:55:45.207440 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:55:45.207420    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43\": container with ID starting with 4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43 not found: ID does not exist" containerID="4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43"
Nov 03 20:55:45.207515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.207459    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43"} err="failed to get container status \"4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43\": rpc error: code = NotFound desc = could not find container \"4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43\": container with ID starting with 4c2a26ec375c1215216fcdf0055cc2d8e3c800b3baba8de46f41f85eaa15ea43 not found: ID does not exist"
Nov 03 20:55:45.256414 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.256390    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f913fa0-4e35-4597-a714-0ef81fe5f7b4" path="/var/lib/kubelet/pods/5f913fa0-4e35-4597-a714-0ef81fe5f7b4/volumes"
Nov 03 20:55:45.368907 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.366874    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"]
Nov 03 20:55:45.432039 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.432003    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15564eb1-595f77d8b-qv9xc"]
Nov 03 20:55:45.640440 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:45.639316    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:46.249768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:46.249737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:46.488597 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:46.488564    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:46.711707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:46.711670    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:46.947744 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:46.947710    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:47.191316 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:47.191280    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:47.258240 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:47.258210    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e98d437-632e-4819-a697-b07c7423bd74" path="/var/lib/kubelet/pods/1e98d437-632e-4819-a697-b07c7423bd74/volumes"
Nov 03 20:55:47.618373 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:47.618314    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:48.255816 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:48.255784    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:48.718361 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:48.717790    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:49.037869 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:49.037785    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:49.280399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:49.273046    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:49.607377 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:49.607317    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:50.011666 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:50.011572    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:51.442627 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.442597    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"]
Nov 03 20:55:51.443034 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.442793    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v" podUID="51c8757b-41f2-475b-822c-411047d525ac" containerName="maas-api" containerID="cri-o://d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c" gracePeriod=30
Nov 03 20:55:51.602730 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.602698    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:51.805453 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.805309    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:51.857197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.857171    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.997083    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca\") pod \"51c8757b-41f2-475b-822c-411047d525ac\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") "
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.997169    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs\") pod \"51c8757b-41f2-475b-822c-411047d525ac\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") "
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.997427    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz\" (UniqueName: \"kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz\") pod \"51c8757b-41f2-475b-822c-411047d525ac\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") "
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.997528    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") pod \"51c8757b-41f2-475b-822c-411047d525ac\" (UID: \"51c8757b-41f2-475b-822c-411047d525ac\") "
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.997795    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca" pod "51c8757b-41f2-475b-822c-411047d525ac" (UID: "51c8757b-41f2-475b-822c-411047d525ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:51.998272    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51c8757b-41f2-475b-822c-411047d525ac-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:52.001593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.001502    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs" pod "51c8757b-41f2-475b-822c-411047d525ac" (UID: "51c8757b-41f2-475b-822c-411047d525ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:52.009489 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.009446    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz" pod "51c8757b-41f2-475b-822c-411047d525ac" (UID: "51c8757b-41f2-475b-822c-411047d525ac"). InnerVolumeSpecName "kube-api-access-nfvbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:52.009726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.009696    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls" pod "51c8757b-41f2-475b-822c-411047d525ac" (UID: "51c8757b-41f2-475b-822c-411047d525ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:52.011288 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.011226    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:52.099786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.099755    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:52.099786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.099783    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51c8757b-41f2-475b-822c-411047d525ac-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:52.099987 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.099795    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvbz\" (UniqueName: \"kubernetes.io/projected/51c8757b-41f2-475b-822c-411047d525ac-kube-api-access-nfvbz\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:52.271783 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.271739    5318 generic.go:356] "Generic (PLEG): container finished" podID="51c8757b-41f2-475b-822c-411047d525ac" containerID="d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c" exitCode=0
Nov 03 20:55:52.271949 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.271828    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"
Nov 03 20:55:52.271949 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.271816    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v" event={"ID":"51c8757b-41f2-475b-822c-411047d525ac","Type":"ContainerDied","Data":"d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c"}
Nov 03 20:55:52.272115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.271948    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v" event={"ID":"51c8757b-41f2-475b-822c-411047d525ac","Type":"ContainerDied","Data":"8e94b20111f8cc2065d691a5d40f5b7f037a0fe78089ce2b89d1d72826b09e81"}
Nov 03 20:55:52.272115 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.271981    5318 scope.go:122] "RemoveContainer" containerID="d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c"
Nov 03 20:55:52.297601 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.297577    5318 scope.go:122] "RemoveContainer" containerID="d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c"
Nov 03 20:55:52.297937 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:55:52.297915    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c\": container with ID starting with d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c not found: ID does not exist" containerID="d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c"
Nov 03 20:55:52.297990 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.297950    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c"} err="failed to get container status \"d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c\": rpc error: code = NotFound desc = could not find container \"d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c\": container with ID starting with d3f5f3a93136539f1f42e9d24db1f00bc1f434f8112794047ca00ef0f9298e5c not found: ID does not exist"
Nov 03 20:55:52.428593 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.428497    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"]
Nov 03 20:55:52.474823 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.474785    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9337cba-86b488c589-n827v"]
Nov 03 20:55:52.888322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:52.888291    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:53.108753 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:53.108724    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:53.257418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:53.257302    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c8757b-41f2-475b-822c-411047d525ac" path="/var/lib/kubelet/pods/51c8757b-41f2-475b-822c-411047d525ac/volumes"
Nov 03 20:55:53.711937 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:53.711869    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:53.927525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:53.927462    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:54.151839 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:54.151802    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:54.391292 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:54.391251    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:55.358376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:55.358270    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:55.562459 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:55.562429    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:55.933939 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:55:55.933907    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:55:56.444726 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:56.444654    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:56.649768 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:56.649737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:59.076748 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:55:59.076707    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:01.587824 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:01.587788    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:01.795517 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:01.795478    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:04.022328 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:04.022290    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:08.340060 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340025    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"]
Nov 03 20:56:08.340662 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340649    5318 cpu_manager.go:414] "containerMap: removing container" podUID="5f913fa0-4e35-4597-a714-0ef81fe5f7b4" containerName="maas-api"
Nov 03 20:56:08.340706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340663    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f913fa0-4e35-4597-a714-0ef81fe5f7b4" containerName="maas-api"
Nov 03 20:56:08.340706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340686    5318 cpu_manager.go:414] "containerMap: removing container" podUID="51c8757b-41f2-475b-822c-411047d525ac" containerName="maas-api"
Nov 03 20:56:08.340706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340691    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c8757b-41f2-475b-822c-411047d525ac" containerName="maas-api"
Nov 03 20:56:08.340706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340702    5318 cpu_manager.go:414] "containerMap: removing container" podUID="1e98d437-632e-4819-a697-b07c7423bd74" containerName="maas-api"
Nov 03 20:56:08.340706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340707    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98d437-632e-4819-a697-b07c7423bd74" containerName="maas-api"
Nov 03 20:56:08.340861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340736    5318 cpu_manager.go:414] "containerMap: removing container" podUID="7237dbc3-a76f-408a-a2ea-1422e417411a" containerName="payload-processing"
Nov 03 20:56:08.340861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340741    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7237dbc3-a76f-408a-a2ea-1422e417411a" containerName="payload-processing"
Nov 03 20:56:08.340861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340818    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f913fa0-4e35-4597-a714-0ef81fe5f7b4" containerName="maas-api"
Nov 03 20:56:08.340861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340831    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c8757b-41f2-475b-822c-411047d525ac" containerName="maas-api"
Nov 03 20:56:08.340861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340837    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e98d437-632e-4819-a697-b07c7423bd74" containerName="maas-api"
Nov 03 20:56:08.340861 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.340847    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="7237dbc3-a76f-408a-a2ea-1422e417411a" containerName="payload-processing"
Nov 03 20:56:08.345254 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.345235    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371128 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371100    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371285    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2wc\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371325    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371451    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371587 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371568    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371641 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371628    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.371743 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.371731    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.411331 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.411297    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:56:08.411786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.411765    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:56:08.411995 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.411979    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:56:08.412113 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.412096    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dddj5\""
Nov 03 20:56:08.473100 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473071    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.473211 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473135    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.473273 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473230    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.473328 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473302    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.473427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473397    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.473503 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473483    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.473630 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.473611    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2wc\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.474221 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.474195    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.475823 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.475794    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.475923 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.475900    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.476153 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.476133    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.476241 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.476223    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.499307 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.499281    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"]
Nov 03 20:56:08.564287 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.564259    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.564495 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.564471    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2wc\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2hdsh\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:08.655852 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.655767    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:08.657841 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:08.657816    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:09.087326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:09.087292    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"]
Nov 03 20:56:09.087483 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:56:09.087292    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e5acd5_a635_45bf_9779_ba6d687e94b2.slice/crio-ee3b6ab3e4407e0b4cc2182178ad4caa956eaa8e9728686e8995451c759d54ec WatchSource:0}: Error finding container ee3b6ab3e4407e0b4cc2182178ad4caa956eaa8e9728686e8995451c759d54ec: Status 404 returned error can't find the container with id ee3b6ab3e4407e0b4cc2182178ad4caa956eaa8e9728686e8995451c759d54ec
Nov 03 20:56:09.418920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:09.418884    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" event={"ID":"74e5acd5-a635-45bf-9779-ba6d687e94b2","Type":"ContainerStarted","Data":"ee3b6ab3e4407e0b4cc2182178ad4caa956eaa8e9728686e8995451c759d54ec"}
Nov 03 20:56:13.156862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:13.156828    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:14.927749 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:14.927702    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:56:14.928134 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:14.927780    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:56:15.470328 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:15.470284    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" event={"ID":"74e5acd5-a635-45bf-9779-ba6d687e94b2","Type":"ContainerStarted","Data":"dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1"}
Nov 03 20:56:15.470887 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:15.470852    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:15.630970 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:15.630918    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" podStartSLOduration=1.7942748069999999 podStartE2EDuration="7.630904199s" totalImagesPullingTime="5.836629392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:15.629243581 +0000 UTC m=+1568.608800424" watchObservedRunningTime="2026-11-03 20:56:15.630904199 +0000 UTC m=+1568.610461038"
Nov 03 20:56:16.334632 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:16.334590    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:16.479063 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:16.479026    5318 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2hdsh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:56:16.479228 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:16.479098    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" podUID="74e5acd5-a635-45bf-9779-ba6d687e94b2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:56:17.676544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:17.676506    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:19.478230 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:19.478197    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 20:56:21.642520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:21.642474    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:21.840562 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:21.840525    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:22.180077 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:22.180019    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:22.519190 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:22.519112    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:25.935632 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:56:25.935598    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:56:26.661840 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:26.661782    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:28.323583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:28.323551    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:31.162920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:31.162873    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:35.590081 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:35.590030    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:35.791922 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:35.791891    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:35.999509 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:35.999416    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:36.315711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:36.315637    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:56:40.663478 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:40.663403    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:40.860619 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:40.860589    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:41.307482 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:41.307453    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:44.066148 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:44.066107    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:44.934106 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:44.934072    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:45.392245 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:45.392213    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:45.828734 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:45.828635    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:49.232034 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:49.231999    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:49.465460 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:49.465427    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:50.547694 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:50.547664    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:50.768279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:50.767967    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:52.454915 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:52.454885    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:52.732653 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:52.732560    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:55.810495 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:56:55.810456    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:55.941375 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:56:55.941323    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:57:00.820466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:00.820425    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:05.421551 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:05.421451    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:06.400304 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:06.400257    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:06.612852 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:06.612804    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:10.419549 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:10.419518    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:12.126312 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:12.126281    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:12.378275 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:12.378202    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:12.949947 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:12.949914    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"]
Nov 03 20:57:12.950118 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:12.950076    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy" containerID="cri-o://9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5" gracePeriod=30
Nov 03 20:57:15.714354 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:15.714251    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:17.546921 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:17.546865    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:17.753929 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:17.753871    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:18.225416 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.225363    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:57:18.235366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235312    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235468 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235422    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235459    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235505    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:18.235559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235543    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235585    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235603    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235705    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235743    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235798    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:18.235875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235808    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c\") pod \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\" (UID: \"b8a818e0-b22b-48c7-ae0d-5db2310cc576\") "
Nov 03 20:57:18.235875 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235817    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:18.236000 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.235908    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:18.236207 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.236167    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:18.236573 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.236554    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.236669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.236578    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.236669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.236592    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.236669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.236606    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.236669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.236619    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.237820 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.237796    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:18.237911 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.237830    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "kube-api-access-xc25c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:18.237953 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.237933    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:18.238017 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.238003    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy" pod "b8a818e0-b22b-48c7-ae0d-5db2310cc576" (UID: "b8a818e0-b22b-48c7-ae0d-5db2310cc576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:18.338245 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.338220    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.338245 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.338243    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.338463 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.338263    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc25c\" (UniqueName: \"kubernetes.io/projected/b8a818e0-b22b-48c7-ae0d-5db2310cc576-kube-api-access-xc25c\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.338463 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.338273    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a818e0-b22b-48c7-ae0d-5db2310cc576-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:18.979968 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.979934    5318 generic.go:356] "Generic (PLEG): container finished" podID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerID="9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5" exitCode=0
Nov 03 20:57:18.980469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.979982    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"
Nov 03 20:57:18.980469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.980026    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" event={"ID":"b8a818e0-b22b-48c7-ae0d-5db2310cc576","Type":"ContainerDied","Data":"9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5"}
Nov 03 20:57:18.980469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.980091    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl" event={"ID":"b8a818e0-b22b-48c7-ae0d-5db2310cc576","Type":"ContainerDied","Data":"3589733447b17d4cdde37040c7dfc9824b4f464a9aacecff3bf0fe1c478543e9"}
Nov 03 20:57:18.980469 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.980124    5318 scope.go:122] "RemoveContainer" containerID="9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5"
Nov 03 20:57:18.991690 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.991674    5318 scope.go:122] "RemoveContainer" containerID="9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5"
Nov 03 20:57:18.991988 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:57:18.991969    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5\": container with ID starting with 9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5 not found: ID does not exist" containerID="9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5"
Nov 03 20:57:18.992031 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:18.991997    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5"} err="failed to get container status \"9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5\": rpc error: code = NotFound desc = could not find container \"9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5\": container with ID starting with 9a16ea0109492b04206f975d47f441331cc075323df6144d2fbb5f3ed01212b5 not found: ID does not exist"
Nov 03 20:57:19.197232 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:19.197195    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"]
Nov 03 20:57:19.202624 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:19.202577    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-9nnnl"]
Nov 03 20:57:19.226974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:19.226938    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:19.257105 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:19.257034    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" path="/var/lib/kubelet/pods/b8a818e0-b22b-48c7-ae0d-5db2310cc576/volumes"
Nov 03 20:57:19.445073 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:19.445042    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:20.529413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:20.529385    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:20.804394 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:20.804287    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:21.199287 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:21.199249    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:21.798320 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:21.798288    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:22.116825 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:22.116772    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:25.435456 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:25.435419    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:25.933727 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:57:25.933695    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:57:26.179640 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:26.179603    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:26.457415 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:26.457377    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:29.497897 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:29.497860    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:29.720579 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:29.720543    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:30.830212 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:30.830169    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:36.038854 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:36.038802    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:36.280585 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:36.280556    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:57:41.347561 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:41.347514    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:43.883813 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:43.883775    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:46.019270 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.019234    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"]
Nov 03 20:57:46.019837 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.019823    5318 cpu_manager.go:414] "containerMap: removing container" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy"
Nov 03 20:57:46.019884 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.019838    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy"
Nov 03 20:57:46.019979 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.019970    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a818e0-b22b-48c7-ae0d-5db2310cc576" containerName="istio-proxy"
Nov 03 20:57:46.022718 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.022699    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.074286 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.074252    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-cc5d0f-serving-cert\""
Nov 03 20:57:46.079660 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.079632    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"]
Nov 03 20:57:46.114158 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.114121    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.114365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.114221    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.114365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.114310    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.114572 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.114553    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqdc7\" (UniqueName: \"kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.216193 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.216159    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqdc7\" (UniqueName: \"kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.216524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.216269    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.216524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.216331    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.216524 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:57:46.216427    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-serving-cert: secret "maas-api-e2e-shared-a-w5-cc5d0f-serving-cert" not found
Nov 03 20:57:46.216524 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.216439    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.216524 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:57:46.216500    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls podName:87415c4c-07b7-4a15-a070-8e1e879d0777 nodeName:}" failed. No retries permitted until 2026-11-03 20:57:46.71647977 +0000 UTC m=+1659.696036600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls") pod "maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" (UID: "87415c4c-07b7-4a15-a070-8e1e879d0777") : secret "maas-api-e2e-shared-a-w5-cc5d0f-serving-cert" not found
Nov 03 20:57:46.217065 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.217042    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.219154 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.219122    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.274392 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.274301    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqdc7\" (UniqueName: \"kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.312708 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.312611    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:46.519793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.519765    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:46.727423 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.727393    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.729846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.729814    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.740427 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.740389    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:46.936244 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.936213    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:46.963563 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:46.963538    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:47.332097 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:47.332065    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"]
Nov 03 20:57:47.454693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:47.454661    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:48.231786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:48.231742    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" event={"ID":"87415c4c-07b7-4a15-a070-8e1e879d0777","Type":"ContainerStarted","Data":"d5e2388b7210dfa177a4982628d988367d1547755d5f6f5348ecef1ec1d59721"}
Nov 03 20:57:48.342281 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:48.342250    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:49.243762 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:49.243724    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" event={"ID":"87415c4c-07b7-4a15-a070-8e1e879d0777","Type":"ContainerStarted","Data":"aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9"}
Nov 03 20:57:49.243971 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:49.243792    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:49.361226 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:49.361114    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" podStartSLOduration=3.429498452 podStartE2EDuration="4.361092492s" totalImagesPullingTime="931.59404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:49.357640151 +0000 UTC m=+1662.337196992" watchObservedRunningTime="2026-11-03 20:57:49.361092492 +0000 UTC m=+1662.340649333"
Nov 03 20:57:51.871762 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:51.871724    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:52.180407 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:52.180312    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:52.418792 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:52.418759    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:52.923455 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:52.923420    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:53.143119 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:53.143085    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:55.258614 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:55.258590    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 20:57:55.934387 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:57:55.934351    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:57:56.681396 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:56.681329    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:57.606383 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:57.605135    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:57.811767 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:57:57.811731    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:01.196730 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:01.196696    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:01.419701 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:01.419646    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:02.735787 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:02.735747    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:02.947928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:02.947888    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:03.922366 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:03.922313    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:04.213783 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:04.213705    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:05.624794 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:05.624754    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:07.317421 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:07.317379    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:07.557219 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:07.557186    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:07.773514 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:07.773478    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:08.210540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:08.210502    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:13.206345 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:13.206307    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:14.383729 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:14.383681    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:14.634776 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:14.634689    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:14.854765 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:14.854737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.347631 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:15.347570    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.675114 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:15.675041    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.901784 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:15.901744    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:18.291527 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:18.291490    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:21.865074 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:21.865030    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:22.068019 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:22.067974    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:22.361458 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:22.361420    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:22.555943 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:22.555910    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.077358 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:23.060784    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.282356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:23.280106    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.477127 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:23.477092    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:24.142161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:24.142125    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:24.339417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:24.339379    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:25.936298 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:25.936259    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:58:28.668575 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:28.668540    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:28.984886 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:28.984792    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.520112 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:29.520076    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.752763 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:29.752731    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:32.844353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:32.844308    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:33.292651 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:33.292553    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:58:33.994982 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:33.994923    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:34.297878 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:34.297800    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:34.510581 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:34.510544    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:34.878326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:34.878288    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:35.194712 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:35.194534    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:35.681652 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:35.681606    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:36.052915 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:36.052833    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:36.408485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:36.408450    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:36.665400 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:36.665263    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:36.891270 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:36.891236    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:37.113583 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:37.113548    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:37.507547 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:37.507433    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:37.814297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:37.814198    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:38.147070 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:38.147038    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:38.728252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:38.728222    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:38.936027 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:38.935989    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:39.331520 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:39.331479    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:41.531534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:41.531482    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:42.726087 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:42.726017    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:43.091874 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.091835    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"]
Nov 03 20:58:43.092091 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.092065    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy" containerID="cri-o://510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14" gracePeriod=30
Nov 03 20:58:43.112525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.112501    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"]
Nov 03 20:58:43.112707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.112687    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy" containerID="cri-o://95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb" gracePeriod=30
Nov 03 20:58:43.137835 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.137808    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"]
Nov 03 20:58:43.138020 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.137992    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy" containerID="cri-o://01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be" gracePeriod=30
Nov 03 20:58:43.334453 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.334412    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:43.572544 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.572464    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:43.915359 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:43.915290    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:44.148163 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:44.148130    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:44.397076 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:44.397041    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:45.726405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.725649    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"]
Nov 03 20:58:45.726405 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.725912    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" containerID="cri-o://032d268b87c520d358f9fe48910b737cafca94d96e40ddd81225944bf512f68b" gracePeriod=30
Nov 03 20:58:45.770365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.767392    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"]
Nov 03 20:58:45.770365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.767660    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy" containerID="cri-o://f8c4653a19e264337c57f327e791d7eebf17e6fcf6db3bfcc847c85d4c528a7c" gracePeriod=30
Nov 03 20:58:45.822356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.814662    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"]
Nov 03 20:58:45.822356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.814947    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy" containerID="cri-o://43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1" gracePeriod=30
Nov 03 20:58:45.822356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.818202    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"]
Nov 03 20:58:45.822356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:45.818495    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy" containerID="cri-o://c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72" gracePeriod=30
Nov 03 20:58:46.592406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:46.592368    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:46.808518 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:46.808480    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:47.018660 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:47.018574    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:48.078849 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.078815    5318 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:58:48.079290 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.078868    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:58:48.436244 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.436218    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:58:48.473413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.473392    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:58:48.477062 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.477040    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:58:48.565934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.565908    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.566064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.565938    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.566064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.565981    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.566064 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566008    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.566233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566072    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.566233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566158    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.566233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566207    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566243    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566275    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566298    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566320    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566353    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566391    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566441    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566516    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.566525 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566522    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566532    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566542    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566587    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566673    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566738    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566751    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566845    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566903    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566934    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566962    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.566999    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567056    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567111    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567143    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567154    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567188    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token\") pod \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\" (UID: \"e15d0a92-f1a6-4d7e-a803-d0ee269529f9\") "
Nov 03 20:58:48.567261 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567228    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token\") pod \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\" (UID: \"1b8cfe95-a734-4615-aad1-910e01c0e9e1\") "
Nov 03 20:58:48.568356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567276    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.568356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567315    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs\") pod \"85f6e609-748a-4985-80af-eee106e9158a\" (UID: \"85f6e609-748a-4985-80af-eee106e9158a\") "
Nov 03 20:58:48.568356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.567886    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.568356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.568193    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.568873 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.568848    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:48.569095 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.569074    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:48.569591 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.569558    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.569693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.569664    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:48.569808 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.569788    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "kube-api-access-sh5hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:48.569934 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.569913    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:48.570223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.570144    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.570607 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.570583    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:48.570914 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.570871    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:48.571181 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.571161    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.571435 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.571302    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.571786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.571750    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:48.572060 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572033    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:48.572191 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572162    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:48.572279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572240    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk" pod "e15d0a92-f1a6-4d7e-a803-d0ee269529f9" (UID: "e15d0a92-f1a6-4d7e-a803-d0ee269529f9"). InnerVolumeSpecName "kube-api-access-9wznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:48.572279 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572262    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6" pod "85f6e609-748a-4985-80af-eee106e9158a" (UID: "85f6e609-748a-4985-80af-eee106e9158a"). InnerVolumeSpecName "kube-api-access-v97q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:48.572831 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572809    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572838    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572851    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f6e609-748a-4985-80af-eee106e9158a-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572859    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572869    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572874    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo" pod "1b8cfe95-a734-4615-aad1-910e01c0e9e1" (UID: "1b8cfe95-a734-4615-aad1-910e01c0e9e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572883    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572892    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.572908 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572902    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97q6\" (UniqueName: \"kubernetes.io/projected/85f6e609-748a-4985-80af-eee106e9158a-kube-api-access-v97q6\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572911    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572920    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wznk\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-kube-api-access-9wznk\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572928    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572936    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572944    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572957    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f6e609-748a-4985-80af-eee106e9158a-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572969    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572977    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572986    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.572994    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573003    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573012    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh5hz\" (UniqueName: \"kubernetes.io/projected/1b8cfe95-a734-4615-aad1-910e01c0e9e1-kube-api-access-sh5hz\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573020    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573033    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfe95-a734-4615-aad1-910e01c0e9e1-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573045    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573055    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573063    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f6e609-748a-4985-80af-eee106e9158a-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.573196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.573071    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15d0a92-f1a6-4d7e-a803-d0ee269529f9-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.674678 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.674609    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8cfe95-a734-4615-aad1-910e01c0e9e1-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.842705 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.842669    5318 generic.go:356] "Generic (PLEG): container finished" podID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerID="510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14" exitCode=0
Nov 03 20:58:48.842883 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.842763    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" event={"ID":"1b8cfe95-a734-4615-aad1-910e01c0e9e1","Type":"ContainerDied","Data":"510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14"}
Nov 03 20:58:48.842883 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.842809    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4" event={"ID":"1b8cfe95-a734-4615-aad1-910e01c0e9e1","Type":"ContainerDied","Data":"312c4046549dfb4a12ff103166d629e68c49bc68bec38c9cc80362b9c5b916aa"}
Nov 03 20:58:48.842883 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.842843    5318 scope.go:122] "RemoveContainer" containerID="510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14"
Nov 03 20:58:48.843050 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.842988    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"
Nov 03 20:58:48.849926 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.849707    5318 generic.go:356] "Generic (PLEG): container finished" podID="85f6e609-748a-4985-80af-eee106e9158a" containerID="95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb" exitCode=0
Nov 03 20:58:48.849926 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.849783    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"
Nov 03 20:58:48.849926 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.849821    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" event={"ID":"85f6e609-748a-4985-80af-eee106e9158a","Type":"ContainerDied","Data":"95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb"}
Nov 03 20:58:48.849926 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.849865    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw" event={"ID":"85f6e609-748a-4985-80af-eee106e9158a","Type":"ContainerDied","Data":"e1a11e17d221f84b48d4b6ac2079d68d52b0e39b19a50b3bf384a6b37e34e898"}
Nov 03 20:58:48.860353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.857010    5318 generic.go:356] "Generic (PLEG): container finished" podID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerID="01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be" exitCode=0
Nov 03 20:58:48.860353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.857062    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" event={"ID":"e15d0a92-f1a6-4d7e-a803-d0ee269529f9","Type":"ContainerDied","Data":"01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be"}
Nov 03 20:58:48.860353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.857097    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m" event={"ID":"e15d0a92-f1a6-4d7e-a803-d0ee269529f9","Type":"ContainerDied","Data":"2ab3816071448616b4d0f4493c9c8d5dc9ac3f5c75c81716e166a4d2717efcd9"}
Nov 03 20:58:48.860353 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.857162    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"
Nov 03 20:58:48.905920 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.905473    5318 scope.go:122] "RemoveContainer" containerID="510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14"
Nov 03 20:58:48.906150 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:48.905958    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14\": container with ID starting with 510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14 not found: ID does not exist" containerID="510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14"
Nov 03 20:58:48.906150 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.906102    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14"} err="failed to get container status \"510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14\": rpc error: code = NotFound desc = could not find container \"510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14\": container with ID starting with 510a160e5488e8ef1187298640ce617ccae6dec1e68e63c12cd13d965d77ff14 not found: ID does not exist"
Nov 03 20:58:48.906150 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.906131    5318 scope.go:122] "RemoveContainer" containerID="95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb"
Nov 03 20:58:48.917851 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.917830    5318 scope.go:122] "RemoveContainer" containerID="95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb"
Nov 03 20:58:48.918180 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:48.918158    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb\": container with ID starting with 95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb not found: ID does not exist" containerID="95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb"
Nov 03 20:58:48.918253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.918187    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb"} err="failed to get container status \"95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb\": rpc error: code = NotFound desc = could not find container \"95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb\": container with ID starting with 95abbdc1730b05a1888275ee7067b8d5ecbfa95e8d722295f9c0947a67bf83bb not found: ID does not exist"
Nov 03 20:58:48.918253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.918204    5318 scope.go:122] "RemoveContainer" containerID="01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be"
Nov 03 20:58:48.931676 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.931232    5318 scope.go:122] "RemoveContainer" containerID="01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be"
Nov 03 20:58:48.931916 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:48.931892    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be\": container with ID starting with 01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be not found: ID does not exist" containerID="01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be"
Nov 03 20:58:48.932327 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:48.932298    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be"} err="failed to get container status \"01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be\": rpc error: code = NotFound desc = could not find container \"01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be\": container with ID starting with 01492d556c044d05abb75e37fb3b2da27ac557617deb7022d183525be814f9be not found: ID does not exist"
Nov 03 20:58:49.007024 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.006988    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"]
Nov 03 20:58:49.014649 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.014615    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-tkt5m"]
Nov 03 20:58:49.161082 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.161052    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"]
Nov 03 20:58:49.165394 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.165363    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-dz7mw"]
Nov 03 20:58:49.255902 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.255843    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f6e609-748a-4985-80af-eee106e9158a" path="/var/lib/kubelet/pods/85f6e609-748a-4985-80af-eee106e9158a/volumes"
Nov 03 20:58:49.256417 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.256401    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" path="/var/lib/kubelet/pods/e15d0a92-f1a6-4d7e-a803-d0ee269529f9/volumes"
Nov 03 20:58:49.266683 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.266660    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"]
Nov 03 20:58:49.272807 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:49.272786    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-2r2n4"]
Nov 03 20:58:50.876152 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:50.876116    5318 generic.go:356] "Generic (PLEG): container finished" podID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerID="f8c4653a19e264337c57f327e791d7eebf17e6fcf6db3bfcc847c85d4c528a7c" exitCode=0
Nov 03 20:58:50.876547 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:50.876203    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" event={"ID":"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35","Type":"ContainerDied","Data":"f8c4653a19e264337c57f327e791d7eebf17e6fcf6db3bfcc847c85d4c528a7c"}
Nov 03 20:58:50.878244 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:50.878217    5318 generic.go:356] "Generic (PLEG): container finished" podID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerID="032d268b87c520d358f9fe48910b737cafca94d96e40ddd81225944bf512f68b" exitCode=0
Nov 03 20:58:50.878376 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:50.878295    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" event={"ID":"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2","Type":"ContainerDied","Data":"032d268b87c520d358f9fe48910b737cafca94d96e40ddd81225944bf512f68b"}
Nov 03 20:58:51.053451 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.053279    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:58:51.117033 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117002    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117186 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117052    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117186 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117112    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117186 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117149    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117215    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117250    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117307    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117534 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117487    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:51.117588 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117533    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117641 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117585    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket\") pod \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\" (UID: \"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35\") "
Nov 03 20:58:51.117775 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.117752    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.118243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.118214    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.118243 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.118245    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.118489 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.118312    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.118560 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.118538    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.118739 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.118716    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.123711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.120632    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:51.123711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.120710    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.123711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.120999    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.123711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.122572    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6" pod "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" (UID: "773466d7-2d8c-4b3f-bd3e-c2f335ef0b35"). InnerVolumeSpecName "kube-api-access-8pph6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.134698 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.134673    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:58:51.195607 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.195577    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:58:51.218909 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.218880    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.218971    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219025    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219061    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219148    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219177    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219232    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219244    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219305    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219354    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219381    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219420    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219495    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.219533 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219514    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219549    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219589    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219673    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219732    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219796    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy\") pod \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\" (UID: \"2f928666-acf3-4f06-bcad-1c4ae20c49c1\") "
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.219824    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs\") pod \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\" (UID: \"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2\") "
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220223    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220440    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220772    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220793    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220806    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220821    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pph6\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-kube-api-access-8pph6\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220835    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220848    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220861    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220874    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220886    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220898    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220910    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.220910    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.221220 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.221193    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.221206    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.221651    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.224300    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.224524    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "kube-api-access-zckw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.224634    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.224728    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.225450    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.226129    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.227588    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.227723    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.227891    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f" pod "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" (UID: "0ea2e484-0d8f-45c7-b03b-5037d4c5dae2"). InnerVolumeSpecName "kube-api-access-s477f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.228912 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.227919    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:51.230515 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.230493    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy" pod "2f928666-acf3-4f06-bcad-1c4ae20c49c1" (UID: "2f928666-acf3-4f06-bcad-1c4ae20c49c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.256620 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.256594    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" path="/var/lib/kubelet/pods/1b8cfe95-a734-4615-aad1-910e01c0e9e1/volumes"
Nov 03 20:58:51.321578 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321497    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321578 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321547    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321578    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321640    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321710    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321793 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321766    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321793    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.321994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321911    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.321994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321946    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.321994 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.321962    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.322772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322000    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert\") pod \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\" (UID: \"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c\") "
Nov 03 20:58:51.322772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322296    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.322772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322512    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:51.322772 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322546    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322821    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322846    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322862    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322875    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322888    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322901    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322914    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322927    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322942    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322956    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.322974 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322969    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322982    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.322996    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.323010    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckw9\" (UniqueName: \"kubernetes.io/projected/2f928666-acf3-4f06-bcad-1c4ae20c49c1-kube-api-access-zckw9\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.323026    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.323039    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.323053    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f928666-acf3-4f06-bcad-1c4ae20c49c1-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.323068    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s477f\" (UniqueName: \"kubernetes.io/projected/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2-kube-api-access-s477f\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.323576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.323082    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f928666-acf3-4f06-bcad-1c4ae20c49c1-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.324252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.324167    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:58:51.328985 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.324295    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:58:51.328985 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.324553    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.329297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.329012    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t" pod "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" (UID: "8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c"). InnerVolumeSpecName "kube-api-access-2rk8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:51.424441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.424409    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.424441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.424437    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rk8t\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-kube-api-access-2rk8t\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.424441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.424449    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.424655 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.424458    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:51.885731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.885656    5318 generic.go:356] "Generic (PLEG): container finished" podID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerID="c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72" exitCode=0
Nov 03 20:58:51.885731 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.885712    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"
Nov 03 20:58:51.886197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.885745    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" event={"ID":"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c","Type":"ContainerDied","Data":"c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72"}
Nov 03 20:58:51.886197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.885822    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk" event={"ID":"8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c","Type":"ContainerDied","Data":"f68f3a7e2683528cffabb49bcfe04a27da4f066cdbf62798167f3b3f3a8e3ab7"}
Nov 03 20:58:51.886197 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.885858    5318 scope.go:122] "RemoveContainer" containerID="c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72"
Nov 03 20:58:51.887233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.887210    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"
Nov 03 20:58:51.887233 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.887223    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw" event={"ID":"773466d7-2d8c-4b3f-bd3e-c2f335ef0b35","Type":"ContainerDied","Data":"e8f28885b4e72419e98b66449ae34ea35fee029aecee6d0ba2a2039904f13971"}
Nov 03 20:58:51.889519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.888906    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"
Nov 03 20:58:51.889519 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.888935    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86" event={"ID":"0ea2e484-0d8f-45c7-b03b-5037d4c5dae2","Type":"ContainerDied","Data":"01ec05d5e633b4352f9547084c5fa0ff309d36b44940394827a5f62b21a2e6e9"}
Nov 03 20:58:51.891603 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.891537    5318 generic.go:356] "Generic (PLEG): container finished" podID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerID="43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1" exitCode=0
Nov 03 20:58:51.891603 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.891578    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"
Nov 03 20:58:51.891778 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.891618    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" event={"ID":"2f928666-acf3-4f06-bcad-1c4ae20c49c1","Type":"ContainerDied","Data":"43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1"}
Nov 03 20:58:51.891778 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.891656    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j" event={"ID":"2f928666-acf3-4f06-bcad-1c4ae20c49c1","Type":"ContainerDied","Data":"48b249065b9cbd255e5846040bac8f9c7867e2d15ec11ae9360741f0331a8821"}
Nov 03 20:58:51.896867 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.896852    5318 scope.go:122] "RemoveContainer" containerID="c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72"
Nov 03 20:58:51.897126 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:51.897109    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72\": container with ID starting with c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72 not found: ID does not exist" containerID="c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72"
Nov 03 20:58:51.897165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.897136    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72"} err="failed to get container status \"c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72\": rpc error: code = NotFound desc = could not find container \"c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72\": container with ID starting with c0f7a66a4d23b3c029bc359c39b09db3b89e5820518d4c0b5fbb75091d0c1f72 not found: ID does not exist"
Nov 03 20:58:51.897165 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.897152    5318 scope.go:122] "RemoveContainer" containerID="f8c4653a19e264337c57f327e791d7eebf17e6fcf6db3bfcc847c85d4c528a7c"
Nov 03 20:58:51.906035 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.906015    5318 scope.go:122] "RemoveContainer" containerID="032d268b87c520d358f9fe48910b737cafca94d96e40ddd81225944bf512f68b"
Nov 03 20:58:51.914251 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.914237    5318 scope.go:122] "RemoveContainer" containerID="43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1"
Nov 03 20:58:51.923051 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.923032    5318 scope.go:122] "RemoveContainer" containerID="43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1"
Nov 03 20:58:51.923321 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:51.923295    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1\": container with ID starting with 43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1 not found: ID does not exist" containerID="43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1"
Nov 03 20:58:51.923401 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:51.923323    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1"} err="failed to get container status \"43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1\": rpc error: code = NotFound desc = could not find container \"43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1\": container with ID starting with 43636b7b92079e85b984aa3aa458166e548b7c0a048d2bab56d0717c34123eb1 not found: ID does not exist"
Nov 03 20:58:52.044350 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.044313    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"]
Nov 03 20:58:52.090247 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.090218    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-2p9vk"]
Nov 03 20:58:52.160559 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.160473    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"]
Nov 03 20:58:52.164532 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.164507    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-5jt7j"]
Nov 03 20:58:52.170655 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.170627    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:52.275084 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.275053    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"]
Nov 03 20:58:52.281419 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.281392    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-698dw"]
Nov 03 20:58:52.388829 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.388790    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"]
Nov 03 20:58:52.398248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:52.398220    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-rjg86"]
Nov 03 20:58:53.256899 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:53.256867    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" path="/var/lib/kubelet/pods/0ea2e484-0d8f-45c7-b03b-5037d4c5dae2/volumes"
Nov 03 20:58:53.257365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:53.257332    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" path="/var/lib/kubelet/pods/2f928666-acf3-4f06-bcad-1c4ae20c49c1/volumes"
Nov 03 20:58:53.257845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:53.257825    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" path="/var/lib/kubelet/pods/773466d7-2d8c-4b3f-bd3e-c2f335ef0b35/volumes"
Nov 03 20:58:53.258696 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:53.258678    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" path="/var/lib/kubelet/pods/8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c/volumes"
Nov 03 20:58:55.933745 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:58:55.933712    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:58:56.395095 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.395068    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:56.610900 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.610862    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:56.763654 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.763579    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"]
Nov 03 20:58:56.764113 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764098    5318 cpu_manager.go:414] "containerMap: removing container" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy"
Nov 03 20:58:56.764155 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764114    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy"
Nov 03 20:58:56.764155 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764133    5318 cpu_manager.go:414] "containerMap: removing container" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy"
Nov 03 20:58:56.764155 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764139    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764155    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764160    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764171    5318 cpu_manager.go:414] "containerMap: removing container" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764176    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764190    5318 cpu_manager.go:414] "containerMap: removing container" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764195    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764203    5318 cpu_manager.go:414] "containerMap: removing container" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764208    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764227    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy"
Nov 03 20:58:56.764250 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764231    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764303    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f6e609-748a-4985-80af-eee106e9158a" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764313    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8cfe95-a734-4615-aad1-910e01c0e9e1" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764322    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed2bf7a-cee5-4149-bc9c-74246b0fcd9c" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764328    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f928666-acf3-4f06-bcad-1c4ae20c49c1" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764351    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15d0a92-f1a6-4d7e-a803-d0ee269529f9" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764360    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea2e484-0d8f-45c7-b03b-5037d4c5dae2" containerName="istio-proxy"
Nov 03 20:58:56.764557 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.764368    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="773466d7-2d8c-4b3f-bd3e-c2f335ef0b35" containerName="istio-proxy"
Nov 03 20:58:56.767766 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.767747    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:56.817615 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.817587    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2e3ea2\""
Nov 03 20:58:56.817838 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.817765    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2e3ea2-dockercfg-hbnpk\""
Nov 03 20:58:56.824081 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.824051    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:56.871386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.871149    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"]
Nov 03 20:58:56.911411 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.911330    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6c5w\" (UniqueName: \"kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w\") pod \"payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:56.911738 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.911707    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:56.913326 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.913300    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"]
Nov 03 20:58:56.918466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.918438    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:56.956576 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:56.955743    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"]
Nov 03 20:58:57.017406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.013320    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.017406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.013487    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:57.017406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.013591    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6c5w\" (UniqueName: \"kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w\") pod \"payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:57.017406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.013659    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p47g\" (UniqueName: \"kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g\") pod \"payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.017406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.014759    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:57.074441 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.074407    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6c5w\" (UniqueName: \"kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w\") pod \"payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:57.103716 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.103685    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:57.117223 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.117185    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.117463 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.117440    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p47g\" (UniqueName: \"kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g\") pod \"payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.118111 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.118090    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.203253 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.203072    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p47g\" (UniqueName: \"kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g\") pod \"payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.234466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.234430    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.483961 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.483931    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"]
Nov 03 20:58:57.486069 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:58:57.486039    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec273859_ab17_4832_bf1b_8b0e3fe4edd4.slice/crio-1636864d98452437aae75dff722e971e57e6026e2e71292d39a80e921d3c7492 WatchSource:0}: Error finding container 1636864d98452437aae75dff722e971e57e6026e2e71292d39a80e921d3c7492: Status 404 returned error can't find the container with id 1636864d98452437aae75dff722e971e57e6026e2e71292d39a80e921d3c7492
Nov 03 20:58:57.620845 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.620824    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"]
Nov 03 20:58:57.621699 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:58:57.621676    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081e81bd_83ae_4f9e_a473_5333d11ea7a8.slice/crio-cde4e89211212931dc0818694f587fd71e6b3ab701e5351a8e5aae77ed11161b WatchSource:0}: Error finding container cde4e89211212931dc0818694f587fd71e6b3ab701e5351a8e5aae77ed11161b: Status 404 returned error can't find the container with id cde4e89211212931dc0818694f587fd71e6b3ab701e5351a8e5aae77ed11161b
Nov 03 20:58:57.936846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.936762    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7" event={"ID":"081e81bd-83ae-4f9e-a473-5333d11ea7a8","Type":"ContainerStarted","Data":"9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4"}
Nov 03 20:58:57.936846 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.936826    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:58:57.937044 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.936849    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7" event={"ID":"081e81bd-83ae-4f9e-a473-5333d11ea7a8","Type":"ContainerStarted","Data":"cde4e89211212931dc0818694f587fd71e6b3ab701e5351a8e5aae77ed11161b"}
Nov 03 20:58:57.938418 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.938385    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6" event={"ID":"ec273859-ab17-4832-bf1b-8b0e3fe4edd4","Type":"ContainerStarted","Data":"0c2278b1e66ef87b465f7102c661d593ba0fa0c48600a12dbcf7771c7411ef30"}
Nov 03 20:58:57.938545 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.938440    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6" event={"ID":"ec273859-ab17-4832-bf1b-8b0e3fe4edd4","Type":"ContainerStarted","Data":"1636864d98452437aae75dff722e971e57e6026e2e71292d39a80e921d3c7492"}
Nov 03 20:58:57.938545 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.938478    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:58:57.947323 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.947300    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"]
Nov 03 20:58:57.947539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:57.947508    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy" containerID="cri-o://b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f" gracePeriod=30
Nov 03 20:58:58.050786 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:58.048192    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7" podStartSLOduration=2.048177757 podStartE2EDuration="2.048177757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:58.04783281 +0000 UTC m=+1731.027389672" watchObservedRunningTime="2026-11-03 20:58:58.048177757 +0000 UTC m=+1731.027734596"
Nov 03 20:58:58.163161 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:58:58.163108    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6" podStartSLOduration=2.163090346 podStartE2EDuration="2.163090346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:58.160550483 +0000 UTC m=+1731.140107324" watchObservedRunningTime="2026-11-03 20:58:58.163090346 +0000 UTC m=+1731.142647187"
Nov 03 20:59:01.319889 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:01.319856    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:01.522386 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:01.522325    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:01.862494 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:01.862448    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:02.336635 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:02.336604    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:02.987748 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:02.987719    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:03.285027 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.281811    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:59:03.427370 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427301    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427377    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427477    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427616    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427673    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427713    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427731    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427780    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.427862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427824    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.428368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427869    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.428368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.427900    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token\") pod \"8e7b6894-4449-4178-ab06-faaf1df1351e\" (UID: \"8e7b6894-4449-4178-ab06-faaf1df1351e\") "
Nov 03 20:59:03.428368 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.428262    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:03.429086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.428613    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.429086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.428636    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7b6894-4449-4178-ab06-faaf1df1351e-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.429086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.428651    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.429086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.428993    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:03.429086 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.429054    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:03.431018 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.430968    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "kube-api-access-8jl97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:03.431760 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.431711    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:03.432005 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.431988    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:03.434306 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.434283    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy" pod "8e7b6894-4449-4178-ab06-faaf1df1351e" (UID: "8e7b6894-4449-4178-ab06-faaf1df1351e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:03.530096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.529999    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jl97\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-kube-api-access-8jl97\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.530096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.530029    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.530096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.530040    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.530096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.530050    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.530096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.530059    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7b6894-4449-4178-ab06-faaf1df1351e-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.530096 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.530067    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7b6894-4449-4178-ab06-faaf1df1351e-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:03.770399 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:03.770365    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:04.002116 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.002071    5318 generic.go:356] "Generic (PLEG): container finished" podID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerID="b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f" exitCode=0
Nov 03 20:59:04.002310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.002171    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" event={"ID":"8e7b6894-4449-4178-ab06-faaf1df1351e","Type":"ContainerDied","Data":"b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f"}
Nov 03 20:59:04.002310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.002206    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"
Nov 03 20:59:04.002310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.002256    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds" event={"ID":"8e7b6894-4449-4178-ab06-faaf1df1351e","Type":"ContainerDied","Data":"4d5f224addd421bb605bb050170d0871d6b03da9a8558b08c607df65cadbdd72"}
Nov 03 20:59:04.002310 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.002289    5318 scope.go:122] "RemoveContainer" containerID="b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f"
Nov 03 20:59:04.016752 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.016733    5318 scope.go:122] "RemoveContainer" containerID="b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f"
Nov 03 20:59:04.017123 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:04.017101    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f\": container with ID starting with b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f not found: ID does not exist" containerID="b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f"
Nov 03 20:59:04.017196 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.017132    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f"} err="failed to get container status \"b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f\": rpc error: code = NotFound desc = could not find container \"b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f\": container with ID starting with b9ac1d7736de2ec3da8b6056e3b47217164e0c7ff7a656f0e493a3a1d89cda2f not found: ID does not exist"
Nov 03 20:59:04.173382 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.169173    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"]
Nov 03 20:59:04.211890 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:04.211855    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-cc7ds"]
Nov 03 20:59:05.257928 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:05.257894    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" path="/var/lib/kubelet/pods/8e7b6894-4449-4178-ab06-faaf1df1351e/volumes"
Nov 03 20:59:06.074623 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:06.074575    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:07.952466 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:07.952373    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:08.185737 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:08.185705    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:08.948025 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:08.947993    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 20:59:08.948186 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:08.948056    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 20:59:11.320362 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.319399    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:11.563365 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.563308    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:11.642889 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.642849    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"]
Nov 03 20:59:11.643424 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.643407    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy"
Nov 03 20:59:11.643424 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.643422    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy"
Nov 03 20:59:11.643579 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.643547    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7b6894-4449-4178-ab06-faaf1df1351e" containerName="istio-proxy"
Nov 03 20:59:11.648888 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.648859    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.709356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.705649    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"]
Nov 03 20:59:11.741137 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.741102    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.741328 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.741151    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5dt\" (UniqueName: \"kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt\") pod \"payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.744073 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.744049    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-9ce672-dockercfg-fb6qm\""
Nov 03 20:59:11.744571 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.744545    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-9ce672\""
Nov 03 20:59:11.792352 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.788859    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:11.820385 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.820353    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"]
Nov 03 20:59:11.824568 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.824547    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:11.842477 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.842442    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.842615 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.842499    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5dt\" (UniqueName: \"kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt\") pod \"payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.843173 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.843151    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.879894 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.879562    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"]
Nov 03 20:59:11.918053 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.917973    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5dt\" (UniqueName: \"kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt\") pod \"payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:11.943587 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.943552    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:11.943773 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.943609    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhms\" (UniqueName: \"kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms\") pod \"payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:11.970162 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:11.968915    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:12.052307 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.052264    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:12.052535 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.052515    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhms\" (UniqueName: \"kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms\") pod \"payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:12.053489 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.053469    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:12.113248 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.113195    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhms\" (UniqueName: \"kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms\") pod \"payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:12.139825 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.139798    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:12.364263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.364238    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"]
Nov 03 20:59:12.365263 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:59:12.365232    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500d5ae6_802c_4a41_a9be_68ce83661b8d.slice/crio-5ba19ca356943fc5e71f1e08b58ec797f5d7ff9a1532785588725aa14793bb0b WatchSource:0}: Error finding container 5ba19ca356943fc5e71f1e08b58ec797f5d7ff9a1532785588725aa14793bb0b: Status 404 returned error can't find the container with id 5ba19ca356943fc5e71f1e08b58ec797f5d7ff9a1532785588725aa14793bb0b
Nov 03 20:59:12.517693 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:12.517615    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"]
Nov 03 20:59:12.521114 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:59:12.521086    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93681b1e_6714_4daa_855e_ca85e7eae2d0.slice/crio-23f9403c890bb9c60c4a90b1174b27c7c5eaf011768b9e607b0c018fe877f093 WatchSource:0}: Error finding container 23f9403c890bb9c60c4a90b1174b27c7c5eaf011768b9e607b0c018fe877f093: Status 404 returned error can't find the container with id 23f9403c890bb9c60c4a90b1174b27c7c5eaf011768b9e607b0c018fe877f093
Nov 03 20:59:13.106482 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.106326    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl" event={"ID":"500d5ae6-802c-4a41-a9be-68ce83661b8d","Type":"ContainerStarted","Data":"1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed"}
Nov 03 20:59:13.106482 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.106431    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:13.106482 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.106456    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl" event={"ID":"500d5ae6-802c-4a41-a9be-68ce83661b8d","Type":"ContainerStarted","Data":"5ba19ca356943fc5e71f1e08b58ec797f5d7ff9a1532785588725aa14793bb0b"}
Nov 03 20:59:13.108284 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.108248    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq" event={"ID":"93681b1e-6714-4daa-855e-ca85e7eae2d0","Type":"ContainerStarted","Data":"9e55ac8bb65cad945bc642c1c63951897c81ea9b2108261c31832b8fa5368edc"}
Nov 03 20:59:13.108413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.108299    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq" event={"ID":"93681b1e-6714-4daa-855e-ca85e7eae2d0","Type":"ContainerStarted","Data":"23f9403c890bb9c60c4a90b1174b27c7c5eaf011768b9e607b0c018fe877f093"}
Nov 03 20:59:13.108413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.108329    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:13.218582 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.218501    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl" podStartSLOduration=2.218483633 podStartE2EDuration="2.218483633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:13.214078741 +0000 UTC m=+1746.193635581" watchObservedRunningTime="2026-11-03 20:59:13.218483633 +0000 UTC m=+1746.198040474"
Nov 03 20:59:13.326322 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:13.326253    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq" podStartSLOduration=2.326234737 podStartE2EDuration="2.326234737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:13.32266972 +0000 UTC m=+1746.302226585" watchObservedRunningTime="2026-11-03 20:59:13.326234737 +0000 UTC m=+1746.305791579"
Nov 03 20:59:16.862711 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:16.862678    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:21.594588 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:21.594551    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:22.805356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:22.805308    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:24.119010 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:24.118979    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 20:59:24.119439 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:24.119051    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 20:59:25.934666 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:25.934631    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 20:59:26.081694 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.081664    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"]
Nov 03 20:59:26.085706 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.085685    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.183895 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.183868    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ba7e83-serving-cert\""
Nov 03 20:59:26.187041 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.186995    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"]
Nov 03 20:59:26.240617 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.240587    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6wh\" (UniqueName: \"kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.240754 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.240651    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.240803 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.240782    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.240860 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.240845    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.342606 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.342555    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6wh\" (UniqueName: \"kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.342789 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.342645    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.342789 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.342744    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.342889 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.342820    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.342946 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:26.342930    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-serving-cert: secret "maas-api-e2e-api-a-ba7e83-serving-cert" not found
Nov 03 20:59:26.343028 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:26.343013    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls podName:56ffbe61-2fdb-4173-ba5d-2c3f1b972545 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:26.842992732 +0000 UTC m=+1759.822549572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls") pod "maas-api-e2e-api-a-ba7e83-6868888869-ntq94" (UID: "56ffbe61-2fdb-4173-ba5d-2c3f1b972545") : secret "maas-api-e2e-api-a-ba7e83-serving-cert" not found
Nov 03 20:59:26.343669 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.343641    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.346252 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.346230    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.356540 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.356500    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:26.400579 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.400555    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6wh\" (UniqueName: \"kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.850009 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.849973    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:26.852325 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:26.852301    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\") pod \"maas-api-e2e-api-a-ba7e83-6868888869-ntq94\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:27.000239 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:27.000209    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:27.374736 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:27.374708    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"]
Nov 03 20:59:27.375827 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:59:27.375799    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ffbe61_2fdb_4173_ba5d_2c3f1b972545.slice/crio-1eabd82995c3d6189244e28fa904776a8fe6288d2e7dc0cd74865e6418062b6d WatchSource:0}: Error finding container 1eabd82995c3d6189244e28fa904776a8fe6288d2e7dc0cd74865e6418062b6d: Status 404 returned error can't find the container with id 1eabd82995c3d6189244e28fa904776a8fe6288d2e7dc0cd74865e6418062b6d
Nov 03 20:59:27.377297 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:27.377279    5318 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:59:27.801413 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:27.801307    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:28.206791 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:28.206757    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94" event={"ID":"56ffbe61-2fdb-4173-ba5d-2c3f1b972545","Type":"ContainerStarted","Data":"1eabd82995c3d6189244e28fa904776a8fe6288d2e7dc0cd74865e6418062b6d"}
Nov 03 20:59:28.847722 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:28.847687    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:29.217642 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:29.217544    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94" event={"ID":"56ffbe61-2fdb-4173-ba5d-2c3f1b972545","Type":"ContainerStarted","Data":"98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21"}
Nov 03 20:59:29.217642 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:29.217634    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:29.341596 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:29.341521    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94" podStartSLOduration=2.428219995 podStartE2EDuration="3.341501403s" totalImagesPullingTime="913.281408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:29.331810689 +0000 UTC m=+1762.311367541" watchObservedRunningTime="2026-11-03 20:59:29.341501403 +0000 UTC m=+1762.321058244"
Nov 03 20:59:31.698695 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:31.698656    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:32.588479 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:32.588446    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:35.230764 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:35.230735    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 20:59:36.401539 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:36.401469    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:41.155263 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:41.155232    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:44.499684 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:44.499651    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 20:59:45.384127 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.384095    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"]
Nov 03 20:59:45.389485 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.389464    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.440052 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.440033    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e427e7-serving-cert\""
Nov 03 20:59:45.490395 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.490367    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"]
Nov 03 20:59:45.504862 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.504829    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.505258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.504879    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.505258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.505024    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.505258 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.505105    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgcm\" (UniqueName: \"kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.606641 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.606609    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgcm\" (UniqueName: \"kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.606777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.606733    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.606777 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.606754    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.606880 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.606846    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.607009 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:45.606955    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-serving-cert: secret "maas-api-e2e-shared-a-w2-e427e7-serving-cert" not found
Nov 03 20:59:45.607057 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:45.607013    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls podName:cdb91df4-f000-4c39-80cb-87c04c11182b nodeName:}" failed. No retries permitted until 2026-11-03 20:59:46.106999054 +0000 UTC m=+1779.086555872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" (UID: "cdb91df4-f000-4c39-80cb-87c04c11182b") : secret "maas-api-e2e-shared-a-w2-e427e7-serving-cert" not found
Nov 03 20:59:45.607347 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.607320    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.609232 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.609216    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.685406 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.682084    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgcm\" (UniqueName: \"kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:45.693356 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.693028    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:45.918135 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:45.918105    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:46.117039 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:46.116996    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:46.119514 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:46.119495    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:46.131585 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:46.131554    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:46.303707 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:46.303682    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:46.677450 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:46.677420    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"]
Nov 03 20:59:46.678892 ip-10-0-135-141 kubenswrapper[5318]: W1103 20:59:46.678850    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb91df4_f000_4c39_80cb_87c04c11182b.slice/crio-15a8ed5acdb2d0417d1ba53395b9c81f570639e5b85c7dcd39ad80ef3f7ea468 WatchSource:0}: Error finding container 15a8ed5acdb2d0417d1ba53395b9c81f570639e5b85c7dcd39ad80ef3f7ea468: Status 404 returned error can't find the container with id 15a8ed5acdb2d0417d1ba53395b9c81f570639e5b85c7dcd39ad80ef3f7ea468
Nov 03 20:59:47.351354 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:47.351301    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" event={"ID":"cdb91df4-f000-4c39-80cb-87c04c11182b","Type":"ContainerStarted","Data":"15a8ed5acdb2d0417d1ba53395b9c81f570639e5b85c7dcd39ad80ef3f7ea468"}
Nov 03 20:59:48.359169 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:48.359137    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" event={"ID":"cdb91df4-f000-4c39-80cb-87c04c11182b","Type":"ContainerStarted","Data":"5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e"}
Nov 03 20:59:48.359582 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:48.359287    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:48.473699 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:48.473632    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" podStartSLOduration=2.538207704 podStartE2EDuration="3.47361411s" totalImagesPullingTime="935.406406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:48.468914496 +0000 UTC m=+1781.448471336" watchObservedRunningTime="2026-11-03 20:59:48.47361411 +0000 UTC m=+1781.453170951"
Nov 03 20:59:50.521040 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:50.520949    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:50.811218 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:50.811137    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:51.012408 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:51.012375    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:54.372987 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:54.372957    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 20:59:55.290436 ip-10-0-135-141 kubenswrapper[5318]: I1103 20:59:55.290274    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:55.933903 ip-10-0-135-141 kubenswrapper[5318]: E1103 20:59:55.933866    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:00:00.007697 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.007657    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:00.232504 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.232467    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"]
Nov 03 21:00:00.237537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.237514    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.250982 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.250958    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"]
Nov 03 21:00:00.255312 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.255293    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.260207 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.260158    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"]
Nov 03 21:00:00.265236 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.265217    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.313800 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.313774    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"]
Nov 03 21:00:00.318603 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.318575    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"]
Nov 03 21:00:00.323084 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.323058    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-gbmj5\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.323199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.323164    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skdp\" (UniqueName: \"kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.323271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.323197    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.323358 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.323324    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wp8\" (UniqueName: \"kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8\") pod \"maas-api-key-cleanup-29895660-gbmj5\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.323427 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.323410    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5nmr\" (UniqueName: \"kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.323492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.323478    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.333300 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.333274    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:00:00.363572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.363547    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"]
Nov 03 21:00:00.424731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.424703    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-gbmj5\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.424839 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.424773    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skdp\" (UniqueName: \"kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.424942 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.424921    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.425064 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.425050    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wp8\" (UniqueName: \"kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8\") pod \"maas-api-key-cleanup-29895660-gbmj5\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.425147 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.425105    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5nmr\" (UniqueName: \"kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.425224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.425208    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.425420 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.425401    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-gbmj5\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.425639 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.425621    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.425811 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.425793    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.482902 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.482875    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wp8\" (UniqueName: \"kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8\") pod \"maas-api-key-cleanup-29895660-gbmj5\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.483045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.482951    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5nmr\" (UniqueName: \"kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.483045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.482998    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skdp\" (UniqueName: \"kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.551665 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.551608    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:00.571196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.571174    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:00.581754 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.581734    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:00.939004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.938979    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"]
Nov 03 21:00:00.940660 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:00:00.940625    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a251633_761d_4ee5_8986_b7d41b8194ee.slice/crio-95c4c8e7c6da9d86dcb9f6c172c7159c2983941a5d3fd99728f1c8ace9cbe083 WatchSource:0}: Error finding container 95c4c8e7c6da9d86dcb9f6c172c7159c2983941a5d3fd99728f1c8ace9cbe083: Status 404 returned error can't find the container with id 95c4c8e7c6da9d86dcb9f6c172c7159c2983941a5d3fd99728f1c8ace9cbe083
Nov 03 21:00:00.987373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.987348    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"]
Nov 03 21:00:00.987731 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:00:00.987706    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fe0b2d_e203_4d54_b9d8_1858d29c18d8.slice/crio-8fb29b9a73d46a5f89dfe9bdd012ee93072cbcf0a593514453ef3037bdedfd17 WatchSource:0}: Error finding container 8fb29b9a73d46a5f89dfe9bdd012ee93072cbcf0a593514453ef3037bdedfd17: Status 404 returned error can't find the container with id 8fb29b9a73d46a5f89dfe9bdd012ee93072cbcf0a593514453ef3037bdedfd17
Nov 03 21:00:00.988294 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:00:00.988266    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce6bc6a_3366_4636_8333_b2bdb92baa95.slice/crio-6835d9edab2a21d48c0ea2160e1298a1180cfb799d5222204e16495044bba55d WatchSource:0}: Error finding container 6835d9edab2a21d48c0ea2160e1298a1180cfb799d5222204e16495044bba55d: Status 404 returned error can't find the container with id 6835d9edab2a21d48c0ea2160e1298a1180cfb799d5222204e16495044bba55d
Nov 03 21:00:00.991604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:00.991584    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"]
Nov 03 21:00:01.450096 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.450058    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" event={"ID":"8a251633-761d-4ee5-8986-b7d41b8194ee","Type":"ContainerStarted","Data":"e856a4ab88fa4cdb407a9896dcfef9035324ae099f5427ea1c913f3c703c2736"}
Nov 03 21:00:01.450565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.450117    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" event={"ID":"8a251633-761d-4ee5-8986-b7d41b8194ee","Type":"ContainerStarted","Data":"95c4c8e7c6da9d86dcb9f6c172c7159c2983941a5d3fd99728f1c8ace9cbe083"}
Nov 03 21:00:01.451561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.451532    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" event={"ID":"9ce6bc6a-3366-4636-8333-b2bdb92baa95","Type":"ContainerStarted","Data":"6835d9edab2a21d48c0ea2160e1298a1180cfb799d5222204e16495044bba55d"}
Nov 03 21:00:01.453392 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.453363    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerStarted","Data":"8e686b3a889baf92e827c81e3690349b8b688834c9cdedc7237de6d5e1c538e7"}
Nov 03 21:00:01.453477 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.453418    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerStarted","Data":"8fb29b9a73d46a5f89dfe9bdd012ee93072cbcf0a593514453ef3037bdedfd17"}
Nov 03 21:00:01.559529 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.559455    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" podStartSLOduration=1.559442306 podStartE2EDuration="1.559442306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:01.556871663 +0000 UTC m=+1794.536428516" watchObservedRunningTime="2026-11-03 21:00:01.559442306 +0000 UTC m=+1794.538999146"
Nov 03 21:00:01.666566 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:01.666512    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" podStartSLOduration=1.666497404 podStartE2EDuration="1.666497404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:01.663294076 +0000 UTC m=+1794.642850917" watchObservedRunningTime="2026-11-03 21:00:01.666497404 +0000 UTC m=+1794.646054257"
Nov 03 21:00:02.473609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.473559    5318 generic.go:356] "Generic (PLEG): container finished" podID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerID="e856a4ab88fa4cdb407a9896dcfef9035324ae099f5427ea1c913f3c703c2736" exitCode=60
Nov 03 21:00:02.474111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.473660    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" event={"ID":"8a251633-761d-4ee5-8986-b7d41b8194ee","Type":"ContainerDied","Data":"e856a4ab88fa4cdb407a9896dcfef9035324ae099f5427ea1c913f3c703c2736"}
Nov 03 21:00:02.474111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.473965    5318 scope.go:122] "RemoveContainer" containerID="e856a4ab88fa4cdb407a9896dcfef9035324ae099f5427ea1c913f3c703c2736"
Nov 03 21:00:02.475618 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.475591    5318 generic.go:356] "Generic (PLEG): container finished" podID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerID="58ac938ecb38fbd77d20a0868a5c3fb2f93a647c19cf941c2eaf3ce96f8eced1" exitCode=60
Nov 03 21:00:02.475753 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.475682    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" event={"ID":"9ce6bc6a-3366-4636-8333-b2bdb92baa95","Type":"ContainerDied","Data":"58ac938ecb38fbd77d20a0868a5c3fb2f93a647c19cf941c2eaf3ce96f8eced1"}
Nov 03 21:00:02.476246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.475961    5318 scope.go:122] "RemoveContainer" containerID="58ac938ecb38fbd77d20a0868a5c3fb2f93a647c19cf941c2eaf3ce96f8eced1"
Nov 03 21:00:02.478039 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.478003    5318 generic.go:356] "Generic (PLEG): container finished" podID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerID="8e686b3a889baf92e827c81e3690349b8b688834c9cdedc7237de6d5e1c538e7" exitCode=60
Nov 03 21:00:02.478099 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.478045    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerDied","Data":"8e686b3a889baf92e827c81e3690349b8b688834c9cdedc7237de6d5e1c538e7"}
Nov 03 21:00:02.478324 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:02.478309    5318 scope.go:122] "RemoveContainer" containerID="8e686b3a889baf92e827c81e3690349b8b688834c9cdedc7237de6d5e1c538e7"
Nov 03 21:00:03.486606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.486564    5318 generic.go:356] "Generic (PLEG): container finished" podID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerID="74cc910f19e8692b32d14b0f8199b31c090bb654113debcc4106d5c44c27372f" exitCode=60
Nov 03 21:00:03.487043 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.486681    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" event={"ID":"8a251633-761d-4ee5-8986-b7d41b8194ee","Type":"ContainerDied","Data":"74cc910f19e8692b32d14b0f8199b31c090bb654113debcc4106d5c44c27372f"}
Nov 03 21:00:03.487043 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.486736    5318 scope.go:122] "RemoveContainer" containerID="e856a4ab88fa4cdb407a9896dcfef9035324ae099f5427ea1c913f3c703c2736"
Nov 03 21:00:03.487043 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.487026    5318 scope.go:122] "RemoveContainer" containerID="74cc910f19e8692b32d14b0f8199b31c090bb654113debcc4106d5c44c27372f"
Nov 03 21:00:03.487385 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:03.487355    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-gbmj5_odh-ai-gateway-infra(8a251633-761d-4ee5-8986-b7d41b8194ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee"
Nov 03 21:00:03.489072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.489034    5318 generic.go:356] "Generic (PLEG): container finished" podID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerID="69bd8940c0e2e0de1cbe001f7f63706d3c61e57fdb86842d17063c9127248125" exitCode=60
Nov 03 21:00:03.489170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.489145    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" event={"ID":"9ce6bc6a-3366-4636-8333-b2bdb92baa95","Type":"ContainerDied","Data":"69bd8940c0e2e0de1cbe001f7f63706d3c61e57fdb86842d17063c9127248125"}
Nov 03 21:00:03.489407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.489388    5318 scope.go:122] "RemoveContainer" containerID="69bd8940c0e2e0de1cbe001f7f63706d3c61e57fdb86842d17063c9127248125"
Nov 03 21:00:03.489689 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:03.489661    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl_odh-ai-gateway-infra(9ce6bc6a-3366-4636-8333-b2bdb92baa95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95"
Nov 03 21:00:03.492018 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.491650    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerStarted","Data":"28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9"}
Nov 03 21:00:03.532258 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:03.532231    5318 scope.go:122] "RemoveContainer" containerID="58ac938ecb38fbd77d20a0868a5c3fb2f93a647c19cf941c2eaf3ce96f8eced1"
Nov 03 21:00:04.500913 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:04.500886    5318 scope.go:122] "RemoveContainer" containerID="74cc910f19e8692b32d14b0f8199b31c090bb654113debcc4106d5c44c27372f"
Nov 03 21:00:04.501388 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:04.501189    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-gbmj5_odh-ai-gateway-infra(8a251633-761d-4ee5-8986-b7d41b8194ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee"
Nov 03 21:00:04.502789 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:04.502764    5318 scope.go:122] "RemoveContainer" containerID="69bd8940c0e2e0de1cbe001f7f63706d3c61e57fdb86842d17063c9127248125"
Nov 03 21:00:04.503049 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:04.503023    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl_odh-ai-gateway-infra(9ce6bc6a-3366-4636-8333-b2bdb92baa95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95"
Nov 03 21:00:04.504590 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:04.504564    5318 generic.go:356] "Generic (PLEG): container finished" podID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerID="28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9" exitCode=60
Nov 03 21:00:04.504679 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:04.504604    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerDied","Data":"28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9"}
Nov 03 21:00:04.504679 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:04.504660    5318 scope.go:122] "RemoveContainer" containerID="8e686b3a889baf92e827c81e3690349b8b688834c9cdedc7237de6d5e1c538e7"
Nov 03 21:00:04.504906 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:04.504879    5318 scope.go:122] "RemoveContainer" containerID="28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9"
Nov 03 21:00:04.505178 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:04.505155    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz_odh-ai-gateway-infra(c5fe0b2d-e203-4d54-b9d8-1858d29c18d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8"
Nov 03 21:00:05.111673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:05.108535    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:05.328608 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:05.328576    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:05.516672 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:05.516604    5318 scope.go:122] "RemoveContainer" containerID="28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9"
Nov 03 21:00:05.517021 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:05.516844    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz_odh-ai-gateway-infra(c5fe0b2d-e203-4d54-b9d8-1858d29c18d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8"
Nov 03 21:00:05.688199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:05.688166    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:06.187759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:06.187602    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:06.421514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:06.421478    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:06.908605 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:06.908575    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:07.182560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:07.182444    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:07.561627 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:07.561547    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:00:07.568601 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:07.568580    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:00:07.675719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:07.675685    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:10.758820 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:10.758784    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:10.972072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:10.972038    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:13.832349 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:13.832302    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:14.091531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:14.091498    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:15.248559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:15.248531    5318 scope.go:122] "RemoveContainer" containerID="69bd8940c0e2e0de1cbe001f7f63706d3c61e57fdb86842d17063c9127248125"
Nov 03 21:00:15.249051 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:15.248641    5318 scope.go:122] "RemoveContainer" containerID="74cc910f19e8692b32d14b0f8199b31c090bb654113debcc4106d5c44c27372f"
Nov 03 21:00:15.728285 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:15.728246    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:16.004429 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.004318    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:16.226183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.226157    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:16.608777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.608748    5318 generic.go:356] "Generic (PLEG): container finished" podID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerID="c3e52c43c5e5993c15be35cda2de8d623a1f2cab2f6f7bd1d76593cdcd025584" exitCode=60
Nov 03 21:00:16.609265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.608839    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" event={"ID":"8a251633-761d-4ee5-8986-b7d41b8194ee","Type":"ContainerDied","Data":"c3e52c43c5e5993c15be35cda2de8d623a1f2cab2f6f7bd1d76593cdcd025584"}
Nov 03 21:00:16.609265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.608915    5318 scope.go:122] "RemoveContainer" containerID="74cc910f19e8692b32d14b0f8199b31c090bb654113debcc4106d5c44c27372f"
Nov 03 21:00:16.609412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.609320    5318 scope.go:122] "RemoveContainer" containerID="c3e52c43c5e5993c15be35cda2de8d623a1f2cab2f6f7bd1d76593cdcd025584"
Nov 03 21:00:16.609734 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:16.609705    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-gbmj5_odh-ai-gateway-infra(8a251633-761d-4ee5-8986-b7d41b8194ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee"
Nov 03 21:00:16.611072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.611046    5318 generic.go:356] "Generic (PLEG): container finished" podID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerID="f1b8c67cca5772799e6c909714fca8e48b39030cb2a7184a11b267506ad592d5" exitCode=60
Nov 03 21:00:16.611186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.611125    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" event={"ID":"9ce6bc6a-3366-4636-8333-b2bdb92baa95","Type":"ContainerDied","Data":"f1b8c67cca5772799e6c909714fca8e48b39030cb2a7184a11b267506ad592d5"}
Nov 03 21:00:16.611481 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.611466    5318 scope.go:122] "RemoveContainer" containerID="f1b8c67cca5772799e6c909714fca8e48b39030cb2a7184a11b267506ad592d5"
Nov 03 21:00:16.611737 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:16.611720    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl_odh-ai-gateway-infra(9ce6bc6a-3366-4636-8333-b2bdb92baa95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95"
Nov 03 21:00:16.623723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:16.623704    5318 scope.go:122] "RemoveContainer" containerID="69bd8940c0e2e0de1cbe001f7f63706d3c61e57fdb86842d17063c9127248125"
Nov 03 21:00:17.252478 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:17.252444    5318 scope.go:122] "RemoveContainer" containerID="28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9"
Nov 03 21:00:17.625354 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:17.624034    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerStarted","Data":"6ea059526e18b7b28b884e1baf10c14cf3f7d9a82209245be8b479f6997b6ab1"}
Nov 03 21:00:17.734356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:17.733714    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"]
Nov 03 21:00:17.846448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:17.846416    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"]
Nov 03 21:00:17.950984 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:17.950395    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:18.050690 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.050665    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:18.088369 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.087470    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca\") pod \"8a251633-761d-4ee5-8986-b7d41b8194ee\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") "
Nov 03 21:00:18.096364 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.092451    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca" pod "8a251633-761d-4ee5-8986-b7d41b8194ee" (UID: "8a251633-761d-4ee5-8986-b7d41b8194ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:18.096364 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.095524    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8\" (UniqueName: \"kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8\") pod \"8a251633-761d-4ee5-8986-b7d41b8194ee\" (UID: \"8a251633-761d-4ee5-8986-b7d41b8194ee\") "
Nov 03 21:00:18.100550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.099802    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a251633-761d-4ee5-8986-b7d41b8194ee-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.121045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.117592    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8" pod "8a251633-761d-4ee5-8986-b7d41b8194ee" (UID: "8a251633-761d-4ee5-8986-b7d41b8194ee"). InnerVolumeSpecName "kube-api-access-47wp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:18.201295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.201261    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca\") pod \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") "
Nov 03 21:00:18.201476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.201457    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp\" (UniqueName: \"kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp\") pod \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\" (UID: \"9ce6bc6a-3366-4636-8333-b2bdb92baa95\") "
Nov 03 21:00:18.201702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.201661    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca" pod "9ce6bc6a-3366-4636-8333-b2bdb92baa95" (UID: "9ce6bc6a-3366-4636-8333-b2bdb92baa95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:18.202142 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.202117    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce6bc6a-3366-4636-8333-b2bdb92baa95-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.202271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.202146    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wp8\" (UniqueName: \"kubernetes.io/projected/8a251633-761d-4ee5-8986-b7d41b8194ee-kube-api-access-47wp8\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.266252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.265881    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp" pod "9ce6bc6a-3366-4636-8333-b2bdb92baa95" (UID: "9ce6bc6a-3366-4636-8333-b2bdb92baa95"). InnerVolumeSpecName "kube-api-access-8skdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:18.303439 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.303414    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skdp\" (UniqueName: \"kubernetes.io/projected/9ce6bc6a-3366-4636-8333-b2bdb92baa95-kube-api-access-8skdp\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.633035 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.633003    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"
Nov 03 21:00:18.633557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.633005    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5" event={"ID":"8a251633-761d-4ee5-8986-b7d41b8194ee","Type":"ContainerDied","Data":"95c4c8e7c6da9d86dcb9f6c172c7159c2983941a5d3fd99728f1c8ace9cbe083"}
Nov 03 21:00:18.633557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.633194    5318 scope.go:122] "RemoveContainer" containerID="c3e52c43c5e5993c15be35cda2de8d623a1f2cab2f6f7bd1d76593cdcd025584"
Nov 03 21:00:18.659372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.659210    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl" event={"ID":"9ce6bc6a-3366-4636-8333-b2bdb92baa95","Type":"ContainerDied","Data":"6835d9edab2a21d48c0ea2160e1298a1180cfb799d5222204e16495044bba55d"}
Nov 03 21:00:18.659372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.659360    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"
Nov 03 21:00:18.664953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.664906    5318 generic.go:356] "Generic (PLEG): container finished" podID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerID="6ea059526e18b7b28b884e1baf10c14cf3f7d9a82209245be8b479f6997b6ab1" exitCode=60
Nov 03 21:00:18.664953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.664950    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerDied","Data":"6ea059526e18b7b28b884e1baf10c14cf3f7d9a82209245be8b479f6997b6ab1"}
Nov 03 21:00:18.666720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.665428    5318 scope.go:122] "RemoveContainer" containerID="6ea059526e18b7b28b884e1baf10c14cf3f7d9a82209245be8b479f6997b6ab1"
Nov 03 21:00:18.666720 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:18.665784    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz_odh-ai-gateway-infra(c5fe0b2d-e203-4d54-b9d8-1858d29c18d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8"
Nov 03 21:00:18.680256 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.680186    5318 scope.go:122] "RemoveContainer" containerID="f1b8c67cca5772799e6c909714fca8e48b39030cb2a7184a11b267506ad592d5"
Nov 03 21:00:18.693065 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.693044    5318 scope.go:122] "RemoveContainer" containerID="28c9e88aa604073ea90eef6098256917d2df6c7ed792463c205cb8a10ed509f9"
Nov 03 21:00:18.745680 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.745651    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"]
Nov 03 21:00:18.793049 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.792957    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"]
Nov 03 21:00:18.800935 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:18.798825    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-gbmj5"]
Nov 03 21:00:19.011478 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.011404    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"]
Nov 03 21:00:19.020550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.020524    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e427e7-29895660-cm4dl"]
Nov 03 21:00:19.256698 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.256667    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" path="/var/lib/kubelet/pods/8a251633-761d-4ee5-8986-b7d41b8194ee/volumes"
Nov 03 21:00:19.257098 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.257084    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" path="/var/lib/kubelet/pods/9ce6bc6a-3366-4636-8333-b2bdb92baa95/volumes"
Nov 03 21:00:19.847772 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.847749    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:19.924585 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.924544    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca\") pod \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") "
Nov 03 21:00:19.924776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.924648    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr\" (UniqueName: \"kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr\") pod \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\" (UID: \"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8\") "
Nov 03 21:00:19.924879 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.924856    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca" pod "c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" (UID: "c5fe0b2d-e203-4d54-b9d8-1858d29c18d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:19.925282 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.925231    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.926718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:19.926699    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr" pod "c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" (UID: "c5fe0b2d-e203-4d54-b9d8-1858d29c18d8"). InnerVolumeSpecName "kube-api-access-h5nmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:20.026662 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:20.026634    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5nmr\" (UniqueName: \"kubernetes.io/projected/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8-kube-api-access-h5nmr\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.687221 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:20.687186    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"
Nov 03 21:00:20.687408 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:20.687220    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz" event={"ID":"c5fe0b2d-e203-4d54-b9d8-1858d29c18d8","Type":"ContainerDied","Data":"8fb29b9a73d46a5f89dfe9bdd012ee93072cbcf0a593514453ef3037bdedfd17"}
Nov 03 21:00:20.687408 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:20.687280    5318 scope.go:122] "RemoveContainer" containerID="6ea059526e18b7b28b884e1baf10c14cf3f7d9a82209245be8b479f6997b6ab1"
Nov 03 21:00:20.869971 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:20.869933    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"]
Nov 03 21:00:20.873902 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:20.873871    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895660-9r6kz"]
Nov 03 21:00:21.257849 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:21.257816    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" path="/var/lib/kubelet/pods/c5fe0b2d-e203-4d54-b9d8-1858d29c18d8/volumes"
Nov 03 21:00:21.303049 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:21.303012    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:22.388196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:22.388162    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:22.591690 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:22.591644    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:23.249332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:23.249293    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:25.935875 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:25.935835    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:00:26.414772 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:26.414739    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:26.428748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:26.428715    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:00:30.139632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:30.139598    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:30.350833 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:30.350791    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:31.784381 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:31.784327    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:36.474039 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:36.473983    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:41.045320 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:41.045288    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:42.965655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:42.965593    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:45.898773 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:45.898728    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:46.314484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:46.314390    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:47.207736 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:47.207701    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:47.883481 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:47.883441    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:48.375263 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:48.375227    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:51.199398 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:51.199367    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:52.829020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:52.828985    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:53.049516 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:53.049473    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:55.934489 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:00:55.934447    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:00:56.002262 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:56.002228    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:57.512735 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:57.512696    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:57.727994 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:00:57.727958    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:00.743332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:00.743299    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:01.052595 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:01.052519    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:05.995491 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:05.995192    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:11.255513 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:11.255469    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:11.469540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:11.469491    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:11.874794 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:11.874738    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:12.279689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:12.279600    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:12.643894 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:12.643844    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:12.855403 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:12.855366    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:12.960398 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:12.960292    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"]
Nov 03 21:01:12.961288 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:12.961245    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy" containerID="cri-o://cd15d080c8ed6a75014df19844c7d70e5fe323c575c101b7bf6edb80c0ff6d38" gracePeriod=30
Nov 03 21:01:16.494045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:16.494004    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"]
Nov 03 21:01:16.494577 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:16.494272    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy" containerID="cri-o://3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b" gracePeriod=30
Nov 03 21:01:16.576847 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:16.576802    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:17.817508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:17.817471    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:18.051221 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.051142    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:18.131720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.131688    5318 generic.go:356] "Generic (PLEG): container finished" podID="e7453bc9-7604-4959-b822-acbf65383db8" containerID="cd15d080c8ed6a75014df19844c7d70e5fe323c575c101b7bf6edb80c0ff6d38" exitCode=0
Nov 03 21:01:18.131720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.131730    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" event={"ID":"e7453bc9-7604-4959-b822-acbf65383db8","Type":"ContainerDied","Data":"cd15d080c8ed6a75014df19844c7d70e5fe323c575c101b7bf6edb80c0ff6d38"}
Nov 03 21:01:18.246801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.246774    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 21:01:18.322411 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322371    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322423    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322450    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322580    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322790 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322658    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322790 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322748    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.322790 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322751    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322948 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322865    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.322948 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322924    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.323053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322934    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.323053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322964    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:18.323053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.322976    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.323212 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323091    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket\") pod \"e7453bc9-7604-4959-b822-acbf65383db8\" (UID: \"e7453bc9-7604-4959-b822-acbf65383db8\") "
Nov 03 21:01:18.323604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323549    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.323740 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323702    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.323740 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323726    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.323874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323741    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453bc9-7604-4959-b822-acbf65383db8-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.323874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323756    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.323874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.323769    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.325065 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.325043    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:18.325185 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.325161    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:18.325244 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.325199    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "kube-api-access-f29wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:18.325305 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.325283    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy" pod "e7453bc9-7604-4959-b822-acbf65383db8" (UID: "e7453bc9-7604-4959-b822-acbf65383db8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:18.425152 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.425114    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29wf\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-kube-api-access-f29wf\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.425152 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.425155    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453bc9-7604-4959-b822-acbf65383db8-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.425352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.425172    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453bc9-7604-4959-b822-acbf65383db8-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:18.425352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:18.425186    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453bc9-7604-4959-b822-acbf65383db8-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:19.139623 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:19.139591    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"
Nov 03 21:01:19.140056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:19.139624    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl" event={"ID":"e7453bc9-7604-4959-b822-acbf65383db8","Type":"ContainerDied","Data":"055e0a7ccfc75607c200bcbca13ea1ce99f6ee06a361d911844887f3f0be9599"}
Nov 03 21:01:19.140056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:19.139685    5318 scope.go:122] "RemoveContainer" containerID="cd15d080c8ed6a75014df19844c7d70e5fe323c575c101b7bf6edb80c0ff6d38"
Nov 03 21:01:19.350213 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:19.350176    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"]
Nov 03 21:01:19.396441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:19.396330    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-zxftl"]
Nov 03 21:01:21.026588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.026550    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:21.246283 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.246254    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:21.259246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.259204    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7453bc9-7604-4959-b822-acbf65383db8" path="/var/lib/kubelet/pods/e7453bc9-7604-4959-b822-acbf65383db8/volumes"
Nov 03 21:01:21.809053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.809029    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 21:01:21.960464 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.960425    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:21.971360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971321    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.971537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971379    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.971537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971497    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.971637 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971536    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.971637 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971620    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.971744 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971649    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.971795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.971748    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.972928 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.972895    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.973057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.973022    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.973129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.973054    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.973189 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.973176    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs\") pod \"6d105468-e95b-484c-8592-d6b84aa21d63\" (UID: \"6d105468-e95b-484c-8592-d6b84aa21d63\") "
Nov 03 21:01:21.973189 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.973179    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:21.978731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.978702    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.978862 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.978844    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:21.978975 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.978907    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:21.979041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.978986    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "kube-api-access-xwm4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:21.979101 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979043    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:21.979656 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979635    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d105468-e95b-484c-8592-d6b84aa21d63-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979656 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979656    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979666    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979681    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwm4j\" (UniqueName: \"kubernetes.io/projected/6d105468-e95b-484c-8592-d6b84aa21d63-kube-api-access-xwm4j\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979690    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d105468-e95b-484c-8592-d6b84aa21d63-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979700    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979714    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.979867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.979724    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:21.980183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:21.980072    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy" pod "6d105468-e95b-484c-8592-d6b84aa21d63" (UID: "6d105468-e95b-484c-8592-d6b84aa21d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:22.080951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.080867    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d105468-e95b-484c-8592-d6b84aa21d63-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:22.162923 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.162883    5318 generic.go:356] "Generic (PLEG): container finished" podID="6d105468-e95b-484c-8592-d6b84aa21d63" containerID="3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b" exitCode=0
Nov 03 21:01:22.163174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.162958    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"
Nov 03 21:01:22.163174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.162962    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" event={"ID":"6d105468-e95b-484c-8592-d6b84aa21d63","Type":"ContainerDied","Data":"3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b"}
Nov 03 21:01:22.163174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.163043    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q" event={"ID":"6d105468-e95b-484c-8592-d6b84aa21d63","Type":"ContainerDied","Data":"fb703808a282e601d498c252b5431dc420bc5c8a2ecd4bf433a5b8eaea1b99eb"}
Nov 03 21:01:22.163174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.163066    5318 scope.go:122] "RemoveContainer" containerID="3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b"
Nov 03 21:01:22.174298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.174281    5318 scope.go:122] "RemoveContainer" containerID="3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b"
Nov 03 21:01:22.174594 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:01:22.174576    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b\": container with ID starting with 3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b not found: ID does not exist" containerID="3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b"
Nov 03 21:01:22.174633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.174602    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b"} err="failed to get container status \"3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b\": rpc error: code = NotFound desc = could not find container \"3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b\": container with ID starting with 3e7d5c1012df888d14a9187942332a7622bde84af602c6724c05cd4f232d246b not found: ID does not exist"
Nov 03 21:01:22.319113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.319084    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"]
Nov 03 21:01:22.365178 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.365148    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-qkv8q"]
Nov 03 21:01:22.759384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:22.759258    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:23.259191 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:23.259157    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" path="/var/lib/kubelet/pods/6d105468-e95b-484c-8592-d6b84aa21d63/volumes"
Nov 03 21:01:23.760548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:23.760514    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:24.192359 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:24.190468    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:24.681920 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:24.681885    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:25.934704 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:01:25.934668    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:01:26.694327 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.694278    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"]
Nov 03 21:01:26.695011 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.694988    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:26.695011 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695011    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695027    5318 cpu_manager.go:414] "containerMap: removing container" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695036    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695053    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695061    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695079    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695086    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695102    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695109    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695135    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695142    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695155    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695162    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695200    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695208    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695233    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695240    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695418    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695435    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695445    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695452    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7453bc9-7604-4959-b822-acbf65383db8" containerName="istio-proxy"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695461    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d105468-e95b-484c-8592-d6b84aa21d63" containerName="istio-proxy"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695471    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695483    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695491    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fe0b2d-e203-4d54-b9d8-1858d29c18d8" containerName="cleanup"
Nov 03 21:01:26.695767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.695501    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:26.706468 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.706443    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.741123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.741096    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.741268 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.741149    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.741363 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.741320    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.741494 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.741436    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455t5\" (UniqueName: \"kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.760617 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.760590    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"]
Nov 03 21:01:26.802718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.802692    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-37c5b0-serving-cert\""
Nov 03 21:01:26.842943 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.842903    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.843099 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.842977    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.843148 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.843116    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.843190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.843162    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455t5\" (UniqueName: \"kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.843630 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:01:26.843609    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-serving-cert: secret "maas-api-e2e-shared-b-w6-37c5b0-serving-cert" not found
Nov 03 21:01:26.843695 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:01:26.843684    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls podName:e66c23a8-cb01-4760-862d-735552c14236 nodeName:}" failed. No retries permitted until 2026-11-03 21:01:27.343663863 +0000 UTC m=+1880.323220703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls") pod "maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" (UID: "e66c23a8-cb01-4760-862d-735552c14236") : secret "maas-api-e2e-shared-b-w6-37c5b0-serving-cert" not found
Nov 03 21:01:26.844641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.844619    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.847158 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.847135    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:26.941389 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:26.941361    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455t5\" (UniqueName: \"kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:27.018359 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:27.018261    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:27.216059 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:27.216030    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:27.350198 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:27.350142    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:27.352753 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:27.352726    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:27.620587 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:27.620511    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:27.995156 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:27.995128    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"]
Nov 03 21:01:27.997192 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:01:27.997163    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66c23a8_cb01_4760_862d_735552c14236.slice/crio-80ee604c6e0fecdc43cee78d36227d0664142f19c685b123e68600e645f84cba WatchSource:0}: Error finding container 80ee604c6e0fecdc43cee78d36227d0664142f19c685b123e68600e645f84cba: Status 404 returned error can't find the container with id 80ee604c6e0fecdc43cee78d36227d0664142f19c685b123e68600e645f84cba
Nov 03 21:01:28.212109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:28.212076    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" event={"ID":"e66c23a8-cb01-4760-862d-735552c14236","Type":"ContainerStarted","Data":"80ee604c6e0fecdc43cee78d36227d0664142f19c685b123e68600e645f84cba"}
Nov 03 21:01:29.220492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:29.220451    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" event={"ID":"e66c23a8-cb01-4760-862d-735552c14236","Type":"ContainerStarted","Data":"b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb"}
Nov 03 21:01:29.220920 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:29.220522    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:29.333465 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:29.333365    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" podStartSLOduration=2.353193732 podStartE2EDuration="3.333348571s" totalImagesPullingTime="980.154839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:29.331075402 +0000 UTC m=+1882.310632232" watchObservedRunningTime="2026-11-03 21:01:29.333348571 +0000 UTC m=+1882.312905402"
Nov 03 21:01:29.436221 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:29.436192    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:29.650642 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:29.650604    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:32.167852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:32.167816    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:32.392001 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:32.391968    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:32.889031 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:32.888996    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:33.098961 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.098918    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:33.209826 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.209721    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"]
Nov 03 21:01:33.210418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.210399    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:33.210511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.210420    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:33.210511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.210437    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:33.210511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.210450    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:33.210697 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.210684    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce6bc6a-3366-4636-8333-b2bdb92baa95" containerName="cleanup"
Nov 03 21:01:33.210741 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.210703    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a251633-761d-4ee5-8986-b7d41b8194ee" containerName="cleanup"
Nov 03 21:01:33.214739 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.214718    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.310142 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.310115    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8448c9\""
Nov 03 21:01:33.310380 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.310328    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8448c9-dockercfg-c9njg\""
Nov 03 21:01:33.317790 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.317487    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"]
Nov 03 21:01:33.326937 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.326894    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.327299 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.327277    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mf7q\" (UniqueName: \"kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q\") pod \"payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.428840 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.428806    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mf7q\" (UniqueName: \"kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q\") pod \"payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.429053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.428958    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.429759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.429732    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.486668 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.486588    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mf7q\" (UniqueName: \"kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q\") pod \"payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.528311 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.528281    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:33.924944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:33.924840    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"]
Nov 03 21:01:33.933896 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:01:33.933865    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d79e7f1_10da_4071_b011_dcdbc254136a.slice/crio-72477f225c9432fae7ec02ab8e2738f31ac4436cb89712b331fee479c11cf15f WatchSource:0}: Error finding container 72477f225c9432fae7ec02ab8e2738f31ac4436cb89712b331fee479c11cf15f: Status 404 returned error can't find the container with id 72477f225c9432fae7ec02ab8e2738f31ac4436cb89712b331fee479c11cf15f
Nov 03 21:01:34.257453 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:34.257359    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8" event={"ID":"7d79e7f1-10da-4071-b011-dcdbc254136a","Type":"ContainerStarted","Data":"71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9"}
Nov 03 21:01:34.257453 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:34.257413    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8" event={"ID":"7d79e7f1-10da-4071-b011-dcdbc254136a","Type":"ContainerStarted","Data":"72477f225c9432fae7ec02ab8e2738f31ac4436cb89712b331fee479c11cf15f"}
Nov 03 21:01:34.257955 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:34.257461    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:34.376430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:34.373913    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8" podStartSLOduration=1.373894334 podStartE2EDuration="1.373894334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:34.367761252 +0000 UTC m=+1887.347318092" watchObservedRunningTime="2026-11-03 21:01:34.373894334 +0000 UTC m=+1887.353451171"
Nov 03 21:01:35.245357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:35.244739    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:01:37.471638 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:37.471596    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:42.104349 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:42.104303    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:45.266506 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:45.266459    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:01:46.981703 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:46.981624    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:47.507413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:47.507375    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:01:51.795649 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:51.795607    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:52.655583 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:52.655549    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:55.934829 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:01:55.934784    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:01:56.824020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:56.823973    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:57.476031 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:57.475988    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:57.677468 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:01:57.677435    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:01.986435 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:01.986392    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:02.198390 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:02.198331    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:03.493059 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:03.493018    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:04.880444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:04.880409    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"]
Nov 03 21:02:04.880859 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:04.880676    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy" containerID="cri-o://3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6" gracePeriod=30
Nov 03 21:02:05.436901 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:05.436864    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:05.645443 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:05.645412    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:06.133425 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.133388    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"]
Nov 03 21:02:06.137906 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.137888    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.233824 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.233791    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-878695-dockercfg-kd592\""
Nov 03 21:02:06.233824 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.233808    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-878695\""
Nov 03 21:02:06.239056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.239027    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"]
Nov 03 21:02:06.315475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.315444    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.315666 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.315599    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrq6\" (UniqueName: \"kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6\") pod \"payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.417143 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.417060    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.417297 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.417205    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrq6\" (UniqueName: \"kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6\") pod \"payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.417853 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.417833    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.475080 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.475047    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrq6\" (UniqueName: \"kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6\") pod \"payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:06.752672 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:06.752581    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:07.124173 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:07.124144    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"]
Nov 03 21:02:07.126229 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:02:07.126194    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37746100_2141_412c_b8d3_43fde7d8d80c.slice/crio-1d3c4e2de1b46cd240328ce32e82d2d0c7f383276afb0af0fc7a1a9ff5ebebb6 WatchSource:0}: Error finding container 1d3c4e2de1b46cd240328ce32e82d2d0c7f383276afb0af0fc7a1a9ff5ebebb6: Status 404 returned error can't find the container with id 1d3c4e2de1b46cd240328ce32e82d2d0c7f383276afb0af0fc7a1a9ff5ebebb6
Nov 03 21:02:07.498474 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:07.498378    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg" event={"ID":"37746100-2141-412c-b8d3-43fde7d8d80c","Type":"ContainerStarted","Data":"85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4"}
Nov 03 21:02:07.498474 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:07.498440    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg" event={"ID":"37746100-2141-412c-b8d3-43fde7d8d80c","Type":"ContainerStarted","Data":"1d3c4e2de1b46cd240328ce32e82d2d0c7f383276afb0af0fc7a1a9ff5ebebb6"}
Nov 03 21:02:07.498474 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:07.498469    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:07.609763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:07.609708    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg" podStartSLOduration=1.609694011 podStartE2EDuration="1.609694011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:07.60714782 +0000 UTC m=+1920.586704661" watchObservedRunningTime="2026-11-03 21:02:07.609694011 +0000 UTC m=+1920.589250850"
Nov 03 21:02:08.694698 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:08.694661    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:10.161876 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.161823    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 21:02:10.268090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268056    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268144    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268169    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268390 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268250    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268390 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268300    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268390 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268347    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268427    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268491    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268535    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket\") pod \"245e503f-5797-4d03-be60-5b8c6ee8f741\" (UID: \"245e503f-5797-4d03-be60-5b8c6ee8f741\") "
Nov 03 21:02:10.268727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268628    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:10.268727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268657    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:10.268876 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.268855    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:10.269063 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.269040    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:10.269216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.269194    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.269216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.269216    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.269434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.269233    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.269434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.269247    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.269434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.269348    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:10.270416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.270394    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:02:10.270619 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.270600    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:10.270796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.270774    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "kube-api-access-rdzhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:10.270987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.270972    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token" pod "245e503f-5797-4d03-be60-5b8c6ee8f741" (UID: "245e503f-5797-4d03-be60-5b8c6ee8f741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:10.370732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.370676    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzhg\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-kube-api-access-rdzhg\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.370732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.370712    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.370732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.370727    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.371025 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.370741    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245e503f-5797-4d03-be60-5b8c6ee8f741-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.371025 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.370753    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245e503f-5797-4d03-be60-5b8c6ee8f741-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:10.521788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.521703    5318 generic.go:356] "Generic (PLEG): container finished" podID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerID="3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6" exitCode=0
Nov 03 21:02:10.521788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.521748    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"
Nov 03 21:02:10.521997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.521787    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" event={"ID":"245e503f-5797-4d03-be60-5b8c6ee8f741","Type":"ContainerDied","Data":"3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6"}
Nov 03 21:02:10.521997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.521843    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg" event={"ID":"245e503f-5797-4d03-be60-5b8c6ee8f741","Type":"ContainerDied","Data":"6f21a99138cad5741b4ebbc3c7a74494d17f0f619fadfcedb1cd66b5738969cf"}
Nov 03 21:02:10.521997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.521866    5318 scope.go:122] "RemoveContainer" containerID="3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6"
Nov 03 21:02:10.532713 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.532693    5318 scope.go:122] "RemoveContainer" containerID="3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6"
Nov 03 21:02:10.532975 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:02:10.532955    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6\": container with ID starting with 3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6 not found: ID does not exist" containerID="3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6"
Nov 03 21:02:10.533074 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.532983    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6"} err="failed to get container status \"3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6\": rpc error: code = NotFound desc = could not find container \"3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6\": container with ID starting with 3be4a0d6eb7b68d7bd2f5135a3210658590b9bdddc55fa545da7a80d07830db6 not found: ID does not exist"
Nov 03 21:02:10.677277 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.677247    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"]
Nov 03 21:02:10.681024 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:10.681000    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-t86gg"]
Nov 03 21:02:11.256414 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:11.256376    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" path="/var/lib/kubelet/pods/245e503f-5797-4d03-be60-5b8c6ee8f741/volumes"
Nov 03 21:02:13.495949 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:13.495833    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:18.506922 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:18.506880    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:18.510910 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:18.510884    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:02:18.803673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:18.803565    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:19.010645 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:19.010604    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:23.469110 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:23.469070    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:24.101811 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:24.101756    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:25.934915 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:02:25.934879    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:02:28.439677 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:28.439591    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:30.847722 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:30.847687    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:02:33.352757 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:33.352722    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:38.150426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:38.150387    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:42.798126 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:42.798088    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:47.434761 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:47.434722    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:47.648779 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:47.648748    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:52.308589 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:52.308551    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:55.935512 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:02:55.935477    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:02:57.420106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:57.420071    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:59.849595 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:02:59.849556    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:00.049129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:00.049101    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:02.430488 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:02.430443    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:02.649368 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:02.649322    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:07.259141 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:07.259112    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:12.248135 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:12.248097    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:17.346397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:17.346306    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:21.509360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:21.508599    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:21.707765 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:21.707733    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:21.925458 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:21.925411    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:24.019638 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:24.019598    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:24.235007 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:24.234964    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:24.498840 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:24.498801    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:25.934261 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:03:25.934220    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:03:26.618419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:26.618373    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:26.788639 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:26.788603    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:03:31.186563 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:31.186525    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:35.872838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:35.872746    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:40.234427 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:40.234380    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:40.443646 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:40.443611    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:45.224228 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:45.224194    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:50.053706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:50.053667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:54.797165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:54.797122    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:55.934429 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:03:55.934389    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:03:59.205260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.205221    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"]
Nov 03 21:03:59.207797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.205560    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" containerID="cri-o://c493d494ad26c1d2be8da87303693804abec49954cf8fbc84fd49fdfebaeae59" gracePeriod=30
Nov 03 21:03:59.262665 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.262637    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"]
Nov 03 21:03:59.263246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.262822    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy" containerID="cri-o://133d88ff99f3947180f10cf8ae34ee6d58f510eebc77a8b242f49edf1adc155c" gracePeriod=30
Nov 03 21:03:59.295847 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.295817    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"]
Nov 03 21:03:59.296080 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.296032    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" containerID="cri-o://87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107" gracePeriod=30
Nov 03 21:03:59.723661 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:03:59.723598    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:01.505988 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:01.505954    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:01.506485 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:01.506026    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:02.748953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:02.748922    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:03.167183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:03.167140    5318 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:03.167379 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:03.167206    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:04.279522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.279486    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:04.382217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.382188    5318 generic.go:356] "Generic (PLEG): container finished" podID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerID="c493d494ad26c1d2be8da87303693804abec49954cf8fbc84fd49fdfebaeae59" exitCode=0
Nov 03 21:04:04.382371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.382253    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" event={"ID":"05e06b7a-a007-44bb-bc50-8e766d772f18","Type":"ContainerDied","Data":"c493d494ad26c1d2be8da87303693804abec49954cf8fbc84fd49fdfebaeae59"}
Nov 03 21:04:04.384457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.384438    5318 generic.go:356] "Generic (PLEG): container finished" podID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerID="133d88ff99f3947180f10cf8ae34ee6d58f510eebc77a8b242f49edf1adc155c" exitCode=0
Nov 03 21:04:04.384557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.384472    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" event={"ID":"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0","Type":"ContainerDied","Data":"133d88ff99f3947180f10cf8ae34ee6d58f510eebc77a8b242f49edf1adc155c"}
Nov 03 21:04:04.496065 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.495738    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:04.575818 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.575787    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 21:04:04.620204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620170    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620216    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620242    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620301    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620376    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620427    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620495    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620542    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620598    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo\") pod \"05e06b7a-a007-44bb-bc50-8e766d772f18\" (UID: \"05e06b7a-a007-44bb-bc50-8e766d772f18\") "
Nov 03 21:04:04.620657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620646    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.620939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.620662    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.621273 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.621246    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.621442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.621425    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e06b7a-a007-44bb-bc50-8e766d772f18-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.621512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.621445    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.621512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.621455    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.621512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.621495    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.621728 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.621535    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.624252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.624009    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:04.624252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.624027    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.624252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.624231    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.625746 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.625711    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz" pod "05e06b7a-a007-44bb-bc50-8e766d772f18" (UID: "05e06b7a-a007-44bb-bc50-8e766d772f18"). InnerVolumeSpecName "kube-api-access-q4hpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.646856 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.646785    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 21:04:04.655727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.655711    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 21:04:04.722072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722042    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.722253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722088    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722132    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.722253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722191    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722277    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722310    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722354    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.722475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722381    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722415    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.722732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722498    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722533    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.722732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722592    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722635    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.722732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722670    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.722732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722707    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs\") pod \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\" (UID: \"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0\") "
Nov 03 21:04:04.723019 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722775    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.723019 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722819    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.723019 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722871    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.723019 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.722911    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert\") pod \"d7da5416-8610-47b5-8b47-92028576c2ad\" (UID: \"d7da5416-8610-47b5-8b47-92028576c2ad\") "
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723508    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723632    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723665    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hpz\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-kube-api-access-q4hpz\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723690    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723707    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723721    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723726    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723735    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723760    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e06b7a-a007-44bb-bc50-8e766d772f18-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723778    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.723836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723813    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.725561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723940    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.725561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.723640    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.725561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.724046    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.725561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.724238    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.725561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.724280    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:04.726880 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.726851    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.727475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.727450    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.727630 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.727605    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:04.728020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.727993    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:04.728020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.727991    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.728386 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.728363    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:04.728545 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.728516    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd" pod "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" (UID: "6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0"). InnerVolumeSpecName "kube-api-access-qvxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.729385 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.729360    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb" pod "d7da5416-8610-47b5-8b47-92028576c2ad" (UID: "d7da5416-8610-47b5-8b47-92028576c2ad"). InnerVolumeSpecName "kube-api-access-h5lnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:04.735260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.735224    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:04.825057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.824975    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.824999    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825013    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825028    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvxcd\" (UniqueName: \"kubernetes.io/projected/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-kube-api-access-qvxcd\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825045    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825059    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825067    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825076    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5lnb\" (UniqueName: \"kubernetes.io/projected/d7da5416-8610-47b5-8b47-92028576c2ad-kube-api-access-h5lnb\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825085    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825098    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825106    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825114    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825123    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825133    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7da5416-8610-47b5-8b47-92028576c2ad-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825141    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7da5416-8610-47b5-8b47-92028576c2ad-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825149    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7da5416-8610-47b5-8b47-92028576c2ad-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:04.825508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:04.825158    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:05.403239 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.403201    5318 generic.go:356] "Generic (PLEG): container finished" podID="d7da5416-8610-47b5-8b47-92028576c2ad" containerID="87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107" exitCode=0
Nov 03 21:04:05.403776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.403411    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" event={"ID":"d7da5416-8610-47b5-8b47-92028576c2ad","Type":"ContainerDied","Data":"87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107"}
Nov 03 21:04:05.403776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.403466    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d" event={"ID":"d7da5416-8610-47b5-8b47-92028576c2ad","Type":"ContainerDied","Data":"3dc80230b1709328ef7605ae2325292eab6dbf1224e5c02dba891f4c2455c4de"}
Nov 03 21:04:05.403776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.403497    5318 scope.go:122] "RemoveContainer" containerID="87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107"
Nov 03 21:04:05.403776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.403640    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"
Nov 03 21:04:05.414787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.410561    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv" event={"ID":"6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0","Type":"ContainerDied","Data":"1e117a50df1074efa90a97598d6c496554c666cca4eb2352f827bad5305bd721"}
Nov 03 21:04:05.414787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.410661    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"
Nov 03 21:04:05.416093 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.415386    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq" event={"ID":"05e06b7a-a007-44bb-bc50-8e766d772f18","Type":"ContainerDied","Data":"82fcc71c3d1139b1bfbf0b72257d0e899faea8c5861d9121cd4b4fc11794c406"}
Nov 03 21:04:05.416093 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.415492    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"
Nov 03 21:04:05.426793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.426770    5318 scope.go:122] "RemoveContainer" containerID="87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107"
Nov 03 21:04:05.427127 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:04:05.427103    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107\": container with ID starting with 87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107 not found: ID does not exist" containerID="87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107"
Nov 03 21:04:05.427228 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.427152    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107"} err="failed to get container status \"87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107\": rpc error: code = NotFound desc = could not find container \"87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107\": container with ID starting with 87843814a418a891612eeaf8256009e0614fe427bd7461983a838e7f79778107 not found: ID does not exist"
Nov 03 21:04:05.427228 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.427178    5318 scope.go:122] "RemoveContainer" containerID="133d88ff99f3947180f10cf8ae34ee6d58f510eebc77a8b242f49edf1adc155c"
Nov 03 21:04:05.436492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.436469    5318 scope.go:122] "RemoveContainer" containerID="c493d494ad26c1d2be8da87303693804abec49954cf8fbc84fd49fdfebaeae59"
Nov 03 21:04:05.561612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.561532    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"]
Nov 03 21:04:05.611003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.610970    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-j824d"]
Nov 03 21:04:05.714189 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.714159    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"]
Nov 03 21:04:05.718465 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.718433    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-gfpvv"]
Nov 03 21:04:05.821806 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.821459    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"]
Nov 03 21:04:05.824808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:05.824773    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-vqlnq"]
Nov 03 21:04:07.256774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:07.256744    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" path="/var/lib/kubelet/pods/05e06b7a-a007-44bb-bc50-8e766d772f18/volumes"
Nov 03 21:04:07.257177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:07.257163    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" path="/var/lib/kubelet/pods/6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0/volumes"
Nov 03 21:04:07.257598 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:07.257584    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" path="/var/lib/kubelet/pods/d7da5416-8610-47b5-8b47-92028576c2ad/volumes"
Nov 03 21:04:09.609039 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:09.609003    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:12.011936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.011900    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"]
Nov 03 21:04:12.012388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.012097    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" podUID="74e5acd5-a635-45bf-9779-ba6d687e94b2" containerName="discovery" containerID="cri-o://dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1" gracePeriod=30
Nov 03 21:04:12.303763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.303739    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 21:04:12.418814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.418782    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.418968 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.418840    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.418968 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.418870    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.418968 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.418945    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.419139 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.419046    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.419139 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.419109    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.419241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.419155    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap\") pod \"74e5acd5-a635-45bf-9779-ba6d687e94b2\" (UID: \"74e5acd5-a635-45bf-9779-ba6d687e94b2\") "
Nov 03 21:04:12.420196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.420162    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:12.421577 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.421536    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:12.421826 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.421793    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:12.421952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.421847    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:12.422388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.422364    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:12.422502 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.422378    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:12.422502 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.422484    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc" pod "74e5acd5-a635-45bf-9779-ba6d687e94b2" (UID: "74e5acd5-a635-45bf-9779-ba6d687e94b2"). InnerVolumeSpecName "kube-api-access-2p2wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:12.467925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.467891    5318 generic.go:356] "Generic (PLEG): container finished" podID="74e5acd5-a635-45bf-9779-ba6d687e94b2" containerID="dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1" exitCode=0
Nov 03 21:04:12.468118 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.467954    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"
Nov 03 21:04:12.468118 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.467980    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" event={"ID":"74e5acd5-a635-45bf-9779-ba6d687e94b2","Type":"ContainerDied","Data":"dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1"}
Nov 03 21:04:12.468118 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.468041    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh" event={"ID":"74e5acd5-a635-45bf-9779-ba6d687e94b2","Type":"ContainerDied","Data":"ee3b6ab3e4407e0b4cc2182178ad4caa956eaa8e9728686e8995451c759d54ec"}
Nov 03 21:04:12.468118 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.468084    5318 scope.go:122] "RemoveContainer" containerID="dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1"
Nov 03 21:04:12.483531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.483482    5318 scope.go:122] "RemoveContainer" containerID="dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1"
Nov 03 21:04:12.484514 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:04:12.484485    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1\": container with ID starting with dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1 not found: ID does not exist" containerID="dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1"
Nov 03 21:04:12.484610 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.484525    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1"} err="failed to get container status \"dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1\": rpc error: code = NotFound desc = could not find container \"dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1\": container with ID starting with dbf7165a679e758163b23268094a6d9bf997833f10b8dcc080d1f2ab5959d1d1 not found: ID does not exist"
Nov 03 21:04:12.501744 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:04:12.501718    5318 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e5acd5_a635_45bf_9779_ba6d687e94b2.slice/crio-ee3b6ab3e4407e0b4cc2182178ad4caa956eaa8e9728686e8995451c759d54ec\": RecentStats: unable to find data in memory cache]"
Nov 03 21:04:12.520631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520576    5318 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-kubeconfig\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.520631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520597    5318 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-ca-configmap\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.520631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520607    5318 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/74e5acd5-a635-45bf-9779-ba6d687e94b2-local-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.520631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520616    5318 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-csr-dns-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.520631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520625    5318 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/74e5acd5-a635-45bf-9779-ba6d687e94b2-cacerts\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.520631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520633    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p2wc\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-kube-api-access-2p2wc\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.520856 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.520641    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74e5acd5-a635-45bf-9779-ba6d687e94b2-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.623669 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.623642    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"]
Nov 03 21:04:12.628848 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:12.628823    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2hdsh"]
Nov 03 21:04:13.257017 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:13.256982    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e5acd5-a635-45bf-9779-ba6d687e94b2" path="/var/lib/kubelet/pods/74e5acd5-a635-45bf-9779-ba6d687e94b2/volumes"
Nov 03 21:04:14.330723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:14.330690    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"]
Nov 03 21:04:14.331207 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:14.330944    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy" containerID="cri-o://e5ed65a2afda5a16ae334e86dacc961ca274bff988964a07043d0ab08f17d821" gracePeriod=30
Nov 03 21:04:14.673802 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:14.673769    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:15.846192 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:15.846155    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:04:18.013721 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:18.013667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:18.256914 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:18.256872    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:18.967609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:18.967572    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:19.531463 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.531430    5318 generic.go:356] "Generic (PLEG): container finished" podID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerID="e5ed65a2afda5a16ae334e86dacc961ca274bff988964a07043d0ab08f17d821" exitCode=0
Nov 03 21:04:19.531851 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.531506    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" event={"ID":"84a6ac6d-0255-4ca4-b53d-526371f74f93","Type":"ContainerDied","Data":"e5ed65a2afda5a16ae334e86dacc961ca274bff988964a07043d0ab08f17d821"}
Nov 03 21:04:19.593049 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.593024    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 21:04:19.607354 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607272    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.607354 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607316    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.607679 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607649    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:19.607806 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607696    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.607806 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607738    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.607935 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607842    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.608036 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608015    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.608159 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608131    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.608268 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.607944    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.608268 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608214    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.608559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608479    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.608646 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608590    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.608731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608670    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs\") pod \"84a6ac6d-0255-4ca4-b53d-526371f74f93\" (UID: \"84a6ac6d-0255-4ca4-b53d-526371f74f93\") "
Nov 03 21:04:19.609020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.608978    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.609796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.609773    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ac6d-0255-4ca4-b53d-526371f74f93-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.609908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.609800    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.609908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.609816    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.609908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.609831    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.609908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.609845    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.610597 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.610560    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:19.611443 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.611422    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "kube-api-access-d4pmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:19.611551 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.611513    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:19.611767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.611752    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token" pod "84a6ac6d-0255-4ca4-b53d-526371f74f93" (UID: "84a6ac6d-0255-4ca4-b53d-526371f74f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:19.711000 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.710975    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4pmt\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-kube-api-access-d4pmt\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.711000 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.710998    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.711138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.711007    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:19.711138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:19.711015    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ac6d-0255-4ca4-b53d-526371f74f93-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:20.546242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.546207    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"
Nov 03 21:04:20.546676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.546215    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84" event={"ID":"84a6ac6d-0255-4ca4-b53d-526371f74f93","Type":"ContainerDied","Data":"88f846949a1ab79cc66f8f30ed62a2f82ca75784ed43f779f3f701cf8c8d2fe1"}
Nov 03 21:04:20.546676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.546422    5318 scope.go:122] "RemoveContainer" containerID="e5ed65a2afda5a16ae334e86dacc961ca274bff988964a07043d0ab08f17d821"
Nov 03 21:04:20.580723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.580688    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:20.747966 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.747938    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"]
Nov 03 21:04:20.794022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.793988    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-5vp84"]
Nov 03 21:04:20.834480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:20.834403    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:21.035490 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.035453    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:21.256540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.256512    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" path="/var/lib/kubelet/pods/84a6ac6d-0255-4ca4-b53d-526371f74f93/volumes"
Nov 03 21:04:21.491851 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.491820    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"]
Nov 03 21:04:21.492365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492349    5318 cpu_manager.go:414] "containerMap: removing container" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492367    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492386    5318 cpu_manager.go:414] "containerMap: removing container" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492392    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492406    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492411    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492418    5318 cpu_manager.go:414] "containerMap: removing container" podUID="74e5acd5-a635-45bf-9779-ba6d687e94b2" containerName="discovery"
Nov 03 21:04:21.492421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492423    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e5acd5-a635-45bf-9779-ba6d687e94b2" containerName="discovery"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492433    5318 cpu_manager.go:414] "containerMap: removing container" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492438    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492471    5318 cpu_manager.go:414] "containerMap: removing container" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492476    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492565    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a6ac6d-0255-4ca4-b53d-526371f74f93" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492574    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="245e503f-5797-4d03-be60-5b8c6ee8f741" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492582    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7da5416-8610-47b5-8b47-92028576c2ad" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492588    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c13e2a6-38b9-4429-aa83-c5b1e0ec60e0" containerName="istio-proxy"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492593    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e5acd5-a635-45bf-9779-ba6d687e94b2" containerName="discovery"
Nov 03 21:04:21.492633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.492598    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e06b7a-a007-44bb-bc50-8e766d772f18" containerName="istio-proxy"
Nov 03 21:04:21.497779 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.497758    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.536591 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.536521    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7r2f\" (UniqueName: \"kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.536692 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.536587    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.536729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.536702    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.546578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.546553    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:21.546965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.546787    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:04:21.595510 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.595480    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"]
Nov 03 21:04:21.638251 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.638219    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7r2f\" (UniqueName: \"kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.638410 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.638270    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.638410 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.638359    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.639101 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.639070    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.640694 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.640674    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.701557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.701527    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7r2f\" (UniqueName: \"kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:21.706709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.706661    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:21.814616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:21.814537    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:22.185759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:22.185733    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"]
Nov 03 21:04:22.187321 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:04:22.187293    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20976a1d_9c35_4140_b1de_2a1a4da3175f.slice/crio-e51e367c3699e321d4cb228ed6997267d05a994e5bf1357feb726ed7144e3412 WatchSource:0}: Error finding container e51e367c3699e321d4cb228ed6997267d05a994e5bf1357feb726ed7144e3412: Status 404 returned error can't find the container with id e51e367c3699e321d4cb228ed6997267d05a994e5bf1357feb726ed7144e3412
Nov 03 21:04:22.561060 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:22.561025    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw" event={"ID":"20976a1d-9c35-4140-b1de-2a1a4da3175f","Type":"ContainerStarted","Data":"ccae702400906690027e65f1d6e8f6c587888f0357337425f06d250d5f829bdc"}
Nov 03 21:04:22.561418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:22.561078    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw" event={"ID":"20976a1d-9c35-4140-b1de-2a1a4da3175f","Type":"ContainerStarted","Data":"e51e367c3699e321d4cb228ed6997267d05a994e5bf1357feb726ed7144e3412"}
Nov 03 21:04:22.674756 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:22.674702    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw" podStartSLOduration=1.674683202 podStartE2EDuration="1.674683202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:22.671025092 +0000 UTC m=+2055.650581931" watchObservedRunningTime="2026-11-03 21:04:22.674683202 +0000 UTC m=+2055.654240042"
Nov 03 21:04:23.568751 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:23.568717    5318 generic.go:356] "Generic (PLEG): container finished" podID="20976a1d-9c35-4140-b1de-2a1a4da3175f" containerID="ccae702400906690027e65f1d6e8f6c587888f0357337425f06d250d5f829bdc" exitCode=0
Nov 03 21:04:23.569217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:23.568812    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw" event={"ID":"20976a1d-9c35-4140-b1de-2a1a4da3175f","Type":"ContainerDied","Data":"ccae702400906690027e65f1d6e8f6c587888f0357337425f06d250d5f829bdc"}
Nov 03 21:04:24.748766 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.748742    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:24.819773 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.819745    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:24.877160 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.877127    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f\" (UniqueName: \"kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f\") pod \"20976a1d-9c35-4140-b1de-2a1a4da3175f\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") "
Nov 03 21:04:24.877376 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.877218    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca\") pod \"20976a1d-9c35-4140-b1de-2a1a4da3175f\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") "
Nov 03 21:04:24.877849 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.877376    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca\") pod \"20976a1d-9c35-4140-b1de-2a1a4da3175f\" (UID: \"20976a1d-9c35-4140-b1de-2a1a4da3175f\") "
Nov 03 21:04:24.877849 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.877806    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca" pod "20976a1d-9c35-4140-b1de-2a1a4da3175f" (UID: "20976a1d-9c35-4140-b1de-2a1a4da3175f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:24.877973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.877960    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:24.879996 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.879963    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f" pod "20976a1d-9c35-4140-b1de-2a1a4da3175f" (UID: "20976a1d-9c35-4140-b1de-2a1a4da3175f"). InnerVolumeSpecName "kube-api-access-x7r2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:24.880267 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.880247    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca" pod "20976a1d-9c35-4140-b1de-2a1a4da3175f" (UID: "20976a1d-9c35-4140-b1de-2a1a4da3175f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:24.979753 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.979681    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7r2f\" (UniqueName: \"kubernetes.io/projected/20976a1d-9c35-4140-b1de-2a1a4da3175f-kube-api-access-x7r2f\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:24.979753 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:24.979716    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20976a1d-9c35-4140-b1de-2a1a4da3175f-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:25.592870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:25.592832    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw" event={"ID":"20976a1d-9c35-4140-b1de-2a1a4da3175f","Type":"ContainerDied","Data":"e51e367c3699e321d4cb228ed6997267d05a994e5bf1357feb726ed7144e3412"}
Nov 03 21:04:25.593104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:25.592846    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"
Nov 03 21:04:25.593104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:25.592931    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e51e367c3699e321d4cb228ed6997267d05a994e5bf1357feb726ed7144e3412"
Nov 03 21:04:25.920599 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:25.920490    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:25.937105 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:04:25.937069    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:04:27.882849 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.882816    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"]
Nov 03 21:04:27.883611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.883589    5318 cpu_manager.go:414] "containerMap: removing container" podUID="20976a1d-9c35-4140-b1de-2a1a4da3175f" containerName="revoke-keys"
Nov 03 21:04:27.883611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.883610    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20976a1d-9c35-4140-b1de-2a1a4da3175f" containerName="revoke-keys"
Nov 03 21:04:27.883808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.883792    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="20976a1d-9c35-4140-b1de-2a1a4da3175f" containerName="revoke-keys"
Nov 03 21:04:27.887357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.887310    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:27.940007 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.939979    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:27.986472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.986444    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:04:27.989592 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:27.989568    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"]
Nov 03 21:04:28.018890 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.018865    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.019008 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.018926    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.019062 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.019046    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6bh\" (UniqueName: \"kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.119834 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.119806    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.119983 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.119879    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.120056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.119984    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6bh\" (UniqueName: \"kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.120623 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.120590    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.122290 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.122267    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.177126 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.177073    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6bh\" (UniqueName: \"kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.200760 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.200733    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:28.287722 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.287632    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:28.580626 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.569319    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"]
Nov 03 21:04:28.616560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:28.616528    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd" event={"ID":"03630754-ba7b-4e10-94a8-9b91daa9b2bf","Type":"ContainerStarted","Data":"207a51701a48fd0984ba4de84249e6ad96b6891369557e63bcfe506e795613a0"}
Nov 03 21:04:29.451263 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:29.451229    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"]
Nov 03 21:04:29.451675 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:29.451470    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" containerID="cri-o://0a320201155b7f0575a00e7ca5cf1f45678631e4230e47e94b0f6718436fc733" gracePeriod=30
Nov 03 21:04:29.624824 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:29.624793    5318 generic.go:356] "Generic (PLEG): container finished" podID="03630754-ba7b-4e10-94a8-9b91daa9b2bf" containerID="7769d32d509b15be0ed1f3a0fc371c33904ac5a972e302a446ade68e5dfc5551" exitCode=0
Nov 03 21:04:29.624986 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:29.624888    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd" event={"ID":"03630754-ba7b-4e10-94a8-9b91daa9b2bf","Type":"ContainerDied","Data":"7769d32d509b15be0ed1f3a0fc371c33904ac5a972e302a446ade68e5dfc5551"}
Nov 03 21:04:30.013958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.013922    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:30.782706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.782684    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:30.850978 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.850950    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca\") pod \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") "
Nov 03 21:04:30.851107 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.851013    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca\") pod \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") "
Nov 03 21:04:30.851209 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.851198    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh\" (UniqueName: \"kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh\") pod \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\" (UID: \"03630754-ba7b-4e10-94a8-9b91daa9b2bf\") "
Nov 03 21:04:30.851308 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.851285    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca" pod "03630754-ba7b-4e10-94a8-9b91daa9b2bf" (UID: "03630754-ba7b-4e10-94a8-9b91daa9b2bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:30.851561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.851543    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.852988 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.852966    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca" pod "03630754-ba7b-4e10-94a8-9b91daa9b2bf" (UID: "03630754-ba7b-4e10-94a8-9b91daa9b2bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:30.853112 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.853096    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh" pod "03630754-ba7b-4e10-94a8-9b91daa9b2bf" (UID: "03630754-ba7b-4e10-94a8-9b91daa9b2bf"). InnerVolumeSpecName "kube-api-access-mn6bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:30.930540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.930331    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:30.953306 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.953267    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03630754-ba7b-4e10-94a8-9b91daa9b2bf-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.953306 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:30.953310    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6bh\" (UniqueName: \"kubernetes.io/projected/03630754-ba7b-4e10-94a8-9b91daa9b2bf-kube-api-access-mn6bh\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:31.638397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:31.638367    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"
Nov 03 21:04:31.638397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:31.638387    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd" event={"ID":"03630754-ba7b-4e10-94a8-9b91daa9b2bf","Type":"ContainerDied","Data":"207a51701a48fd0984ba4de84249e6ad96b6891369557e63bcfe506e795613a0"}
Nov 03 21:04:31.638619 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:31.638445    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207a51701a48fd0984ba4de84249e6ad96b6891369557e63bcfe506e795613a0"
Nov 03 21:04:33.273442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:33.273405    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:33.527466 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:33.527382    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:34.198653 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.198618    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:34.198980 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.198688    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:34.663261 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.663224    5318 generic.go:356] "Generic (PLEG): container finished" podID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerID="0a320201155b7f0575a00e7ca5cf1f45678631e4230e47e94b0f6718436fc733" exitCode=0
Nov 03 21:04:34.663681 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.663311    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" event={"ID":"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c","Type":"ContainerDied","Data":"0a320201155b7f0575a00e7ca5cf1f45678631e4230e47e94b0f6718436fc733"}
Nov 03 21:04:34.728549 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.728488    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 21:04:34.901787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.901751    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.901958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.901832    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.901958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.901914    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.901958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.901944    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.902142 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902006    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.902142 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902039    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.902252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902143    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.902252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902182    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.902252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902240    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.902437 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902300    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token\") pod \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\" (UID: \"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c\") "
Nov 03 21:04:34.902881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902607    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.903048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902927    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.903048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.902951    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:34.903267 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.903062    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:34.903267 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.903156    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.903445 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.903383    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.904641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.904613    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:34.906132 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.906097    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:34.906248 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.906214    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "kube-api-access-7qdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:34.907069 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:34.907048    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token" pod "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" (UID: "9156fa97-8691-42d5-bc3e-1b4b0ad3db1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003893    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003920    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003930    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003939    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdgw\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-kube-api-access-7qdgw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003949    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003959    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.003959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.003968    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:35.303502 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.303470    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:35.515970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.515901    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:35.673538 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.673505    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"
Nov 03 21:04:35.674035 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.673539    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg" event={"ID":"9156fa97-8691-42d5-bc3e-1b4b0ad3db1c","Type":"ContainerDied","Data":"d65ba469fde91f669c1a89a6a987e84a75d91de27307f9188655aee33f271f6e"}
Nov 03 21:04:35.674035 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.673607    5318 scope.go:122] "RemoveContainer" containerID="0a320201155b7f0575a00e7ca5cf1f45678631e4230e47e94b0f6718436fc733"
Nov 03 21:04:35.727287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.727259    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:35.881006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.880972    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"]
Nov 03 21:04:35.925780 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:35.925753    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-6hkgg"]
Nov 03 21:04:37.256116 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:37.256081    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" path="/var/lib/kubelet/pods/9156fa97-8691-42d5-bc3e-1b4b0ad3db1c/volumes"
Nov 03 21:04:39.390567 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:39.390533    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"]
Nov 03 21:04:39.499908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:39.499875    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"]
Nov 03 21:04:39.832606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:39.832512    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:40.035411 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:40.035378    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:40.501299 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:40.501269    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:44.390936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.390878    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7" podUID="081e81bd-83ae-4f9e-a473-5333d11ea7a8" containerName="payload-processing" containerID="cri-o://9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4" gracePeriod=25
Nov 03 21:04:44.500265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.500180    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6" podUID="ec273859-ab17-4832-bf1b-8b0e3fe4edd4" containerName="payload-pre-processing" containerID="cri-o://0c2278b1e66ef87b465f7102c661d593ba0fa0c48600a12dbcf7771c7411ef30" gracePeriod=25
Nov 03 21:04:44.692017 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.691650    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 21:04:44.731332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.731299    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g\" (UniqueName: \"kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g\") pod \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") "
Nov 03 21:04:44.731522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.731447    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume\") pod \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\" (UID: \"081e81bd-83ae-4f9e-a473-5333d11ea7a8\") "
Nov 03 21:04:44.732448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.732420    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume" pod "081e81bd-83ae-4f9e-a473-5333d11ea7a8" (UID: "081e81bd-83ae-4f9e-a473-5333d11ea7a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:44.738169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.738137    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g" pod "081e81bd-83ae-4f9e-a473-5333d11ea7a8" (UID: "081e81bd-83ae-4f9e-a473-5333d11ea7a8"). InnerVolumeSpecName "kube-api-access-2p47g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:44.739026 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.738996    5318 generic.go:356] "Generic (PLEG): container finished" podID="081e81bd-83ae-4f9e-a473-5333d11ea7a8" containerID="9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4" exitCode=0
Nov 03 21:04:44.739162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.739140    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7" event={"ID":"081e81bd-83ae-4f9e-a473-5333d11ea7a8","Type":"ContainerDied","Data":"9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4"}
Nov 03 21:04:44.739250 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.739235    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7" event={"ID":"081e81bd-83ae-4f9e-a473-5333d11ea7a8","Type":"ContainerDied","Data":"cde4e89211212931dc0818694f587fd71e6b3ab701e5351a8e5aae77ed11161b"}
Nov 03 21:04:44.739313 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.739270    5318 scope.go:122] "RemoveContainer" containerID="9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4"
Nov 03 21:04:44.739452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.739436    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"
Nov 03 21:04:44.741617 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.741595    5318 generic.go:356] "Generic (PLEG): container finished" podID="ec273859-ab17-4832-bf1b-8b0e3fe4edd4" containerID="0c2278b1e66ef87b465f7102c661d593ba0fa0c48600a12dbcf7771c7411ef30" exitCode=0
Nov 03 21:04:44.741762 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.741724    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6" event={"ID":"ec273859-ab17-4832-bf1b-8b0e3fe4edd4","Type":"ContainerDied","Data":"0c2278b1e66ef87b465f7102c661d593ba0fa0c48600a12dbcf7771c7411ef30"}
Nov 03 21:04:44.758889 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.758869    5318 scope.go:122] "RemoveContainer" containerID="9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4"
Nov 03 21:04:44.759152 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:04:44.759133    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4\": container with ID starting with 9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4 not found: ID does not exist" containerID="9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4"
Nov 03 21:04:44.759221 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.759159    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4"} err="failed to get container status \"9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4\": rpc error: code = NotFound desc = could not find container \"9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4\": container with ID starting with 9b9048f015cdff8406299c52bcaba8c5b737cf7cb542033b077a9ecee983afe4 not found: ID does not exist"
Nov 03 21:04:44.786906 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.786885    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 21:04:44.841366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.833913    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume\") pod \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") "
Nov 03 21:04:44.841366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.834080    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w\" (UniqueName: \"kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w\") pod \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\" (UID: \"ec273859-ab17-4832-bf1b-8b0e3fe4edd4\") "
Nov 03 21:04:44.841366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.834821    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p47g\" (UniqueName: \"kubernetes.io/projected/081e81bd-83ae-4f9e-a473-5333d11ea7a8-kube-api-access-2p47g\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:44.841366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.834844    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/081e81bd-83ae-4f9e-a473-5333d11ea7a8-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:44.841366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.835487    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume" pod "ec273859-ab17-4832-bf1b-8b0e3fe4edd4" (UID: "ec273859-ab17-4832-bf1b-8b0e3fe4edd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:44.841366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.837365    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w" pod "ec273859-ab17-4832-bf1b-8b0e3fe4edd4" (UID: "ec273859-ab17-4832-bf1b-8b0e3fe4edd4"). InnerVolumeSpecName "kube-api-access-x6c5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:44.936081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.936003    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6c5w\" (UniqueName: \"kubernetes.io/projected/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-kube-api-access-x6c5w\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:44.936081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.936031    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec273859-ab17-4832-bf1b-8b0e3fe4edd4-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:44.991311 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.991269    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"]
Nov 03 21:04:44.994213 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:44.994185    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e3ea2-7bbb5cd6c5-kfvd7"]
Nov 03 21:04:45.060957 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.060902    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:45.259408 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.259284    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081e81bd-83ae-4f9e-a473-5333d11ea7a8" path="/var/lib/kubelet/pods/081e81bd-83ae-4f9e-a473-5333d11ea7a8/volumes"
Nov 03 21:04:45.285699 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.285664    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:45.750190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.750160    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"
Nov 03 21:04:45.750836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.750193    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6" event={"ID":"ec273859-ab17-4832-bf1b-8b0e3fe4edd4","Type":"ContainerDied","Data":"1636864d98452437aae75dff722e971e57e6026e2e71292d39a80e921d3c7492"}
Nov 03 21:04:45.750836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.750253    5318 scope.go:122] "RemoveContainer" containerID="0c2278b1e66ef87b465f7102c661d593ba0fa0c48600a12dbcf7771c7411ef30"
Nov 03 21:04:45.915358 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.915178    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"]
Nov 03 21:04:45.960696 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:45.960665    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e3ea2-6b476996dc-svwr6"]
Nov 03 21:04:47.006449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:47.006416    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:47.227074 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:47.227045    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:47.255913 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:47.255885    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec273859-ab17-4832-bf1b-8b0e3fe4edd4" path="/var/lib/kubelet/pods/ec273859-ab17-4832-bf1b-8b0e3fe4edd4/volumes"
Nov 03 21:04:47.444974 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:47.444937    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:48.084028 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:48.084002    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:51.930313 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:51.930275    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:53.849282 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:53.849245    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:54.078322 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:54.078280    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:54.285462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:54.285432    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:55.933647 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:04:55.933613    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:04:56.997242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:04:56.997209    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:01.573385 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:01.573329    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:04.432085 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:04.432051    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:06.097203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:06.097167    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:06.626164 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:06.626121    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:07.268767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:07.268723    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:07.496707 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:07.496667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:07.609078 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:07.609051    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:05:07.618791 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:07.618769    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:05:10.848930 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:10.848897    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:11.641887 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:11.641857    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:12.029977 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:12.029905    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:14.932763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:14.932727    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:15.153098 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.153057    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:15.300115 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300023    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"]
Nov 03 21:05:15.300615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300598    5318 cpu_manager.go:414] "containerMap: removing container" podUID="081e81bd-83ae-4f9e-a473-5333d11ea7a8" containerName="payload-processing"
Nov 03 21:05:15.300615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300614    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081e81bd-83ae-4f9e-a473-5333d11ea7a8" containerName="payload-processing"
Nov 03 21:05:15.300767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300625    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy"
Nov 03 21:05:15.300767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300630    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy"
Nov 03 21:05:15.300767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300648    5318 cpu_manager.go:414] "containerMap: removing container" podUID="03630754-ba7b-4e10-94a8-9b91daa9b2bf" containerName="revoke-keys"
Nov 03 21:05:15.300767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300653    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03630754-ba7b-4e10-94a8-9b91daa9b2bf" containerName="revoke-keys"
Nov 03 21:05:15.300767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300673    5318 cpu_manager.go:414] "containerMap: removing container" podUID="ec273859-ab17-4832-bf1b-8b0e3fe4edd4" containerName="payload-pre-processing"
Nov 03 21:05:15.300767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300679    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec273859-ab17-4832-bf1b-8b0e3fe4edd4" containerName="payload-pre-processing"
Nov 03 21:05:15.301076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300771    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="03630754-ba7b-4e10-94a8-9b91daa9b2bf" containerName="revoke-keys"
Nov 03 21:05:15.301076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300782    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec273859-ab17-4832-bf1b-8b0e3fe4edd4" containerName="payload-pre-processing"
Nov 03 21:05:15.301076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300793    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="081e81bd-83ae-4f9e-a473-5333d11ea7a8" containerName="payload-processing"
Nov 03 21:05:15.301076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.300798    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9156fa97-8691-42d5-bc3e-1b4b0ad3db1c" containerName="istio-proxy"
Nov 03 21:05:15.306264 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.306243    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.355422 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.355392    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-df71d7\""
Nov 03 21:05:15.437748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.413782    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-df71d7-dockercfg-7z6kv\""
Nov 03 21:05:15.437748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.430609    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"]
Nov 03 21:05:15.437748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.430687    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"]
Nov 03 21:05:15.437748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.435777    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbx69\" (UniqueName: \"kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69\") pod \"payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.437748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.435911    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.437748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.437738    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.488228 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.488195    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"]
Nov 03 21:05:15.536882 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.536845    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.537046 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.537030    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbx69\" (UniqueName: \"kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69\") pod \"payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.537099 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.537084    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqgh\" (UniqueName: \"kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh\") pod \"payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.537144 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.537126    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.537657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.537634    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.578644 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.578565    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:15.594790 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.594763    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbx69\" (UniqueName: \"kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69\") pod \"payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.627424 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.620121    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:15.640549 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.640128    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqgh\" (UniqueName: \"kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh\") pod \"payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.640549 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.640181    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.641210 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.641187    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.716147 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.704011    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqgh\" (UniqueName: \"kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh\") pod \"payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:15.772410 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:15.772374    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:16.027062 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.027034    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"]
Nov 03 21:05:16.028475 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:05:16.028448    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb9d38e_07f3_4993_82ee_732d30c16095.slice/crio-af7c9b4015a7fd0351b4504e64931629dd55e454ecb24977afdaf4f39538e672 WatchSource:0}: Error finding container af7c9b4015a7fd0351b4504e64931629dd55e454ecb24977afdaf4f39538e672: Status 404 returned error can't find the container with id af7c9b4015a7fd0351b4504e64931629dd55e454ecb24977afdaf4f39538e672
Nov 03 21:05:16.178788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.178757    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"]
Nov 03 21:05:16.180892 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:05:16.180854    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e99ba1_7cab_4ebc_b6f5_0ebd07fac0ec.slice/crio-4937281b20f1be05f4f96406c76ef650d2fe98d6372a48e3be8812fd13d48913 WatchSource:0}: Error finding container 4937281b20f1be05f4f96406c76ef650d2fe98d6372a48e3be8812fd13d48913: Status 404 returned error can't find the container with id 4937281b20f1be05f4f96406c76ef650d2fe98d6372a48e3be8812fd13d48913
Nov 03 21:05:16.215460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.215434    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"]
Nov 03 21:05:16.611466 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.611430    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:16.824601 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.824566    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:16.974879 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.974786    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2" event={"ID":"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec","Type":"ContainerStarted","Data":"4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4"}
Nov 03 21:05:16.974879 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.974865    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:16.975101 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.974890    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2" event={"ID":"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec","Type":"ContainerStarted","Data":"4937281b20f1be05f4f96406c76ef650d2fe98d6372a48e3be8812fd13d48913"}
Nov 03 21:05:16.988728 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.988694    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" event={"ID":"3eb9d38e-07f3-4993-82ee-732d30c16095","Type":"ContainerStarted","Data":"3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e"}
Nov 03 21:05:16.989726 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.989706    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:16.989914 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:16.989898    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" event={"ID":"3eb9d38e-07f3-4993-82ee-732d30c16095","Type":"ContainerStarted","Data":"af7c9b4015a7fd0351b4504e64931629dd55e454ecb24977afdaf4f39538e672"}
Nov 03 21:05:17.084537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:17.084486    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2" podStartSLOduration=2.08447116 podStartE2EDuration="2.08447116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:17.083325189 +0000 UTC m=+2110.062882029" watchObservedRunningTime="2026-11-03 21:05:17.08447116 +0000 UTC m=+2110.064028000"
Nov 03 21:05:17.103978 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:17.103942    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:17.194102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:17.194035    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" podStartSLOduration=2.194015954 podStartE2EDuration="2.194015954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:17.192619049 +0000 UTC m=+2110.172175881" watchObservedRunningTime="2026-11-03 21:05:17.194015954 +0000 UTC m=+2110.173572795"
Nov 03 21:05:17.560651 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:17.560600    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:17.792728 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:17.792692    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:21.216443 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.216381    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg" podUID="37746100-2141-412c-b8d3-43fde7d8d80c" containerName="payload-pre-processing" containerID="cri-o://85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4" gracePeriod=25
Nov 03 21:05:21.477480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.477419    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:05:21.513058 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.513022    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6\" (UniqueName: \"kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6\") pod \"37746100-2141-412c-b8d3-43fde7d8d80c\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") "
Nov 03 21:05:21.513231 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.513176    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume\") pod \"37746100-2141-412c-b8d3-43fde7d8d80c\" (UID: \"37746100-2141-412c-b8d3-43fde7d8d80c\") "
Nov 03 21:05:21.514007 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.513973    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume" pod "37746100-2141-412c-b8d3-43fde7d8d80c" (UID: "37746100-2141-412c-b8d3-43fde7d8d80c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:21.516224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.516195    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6" pod "37746100-2141-412c-b8d3-43fde7d8d80c" (UID: "37746100-2141-412c-b8d3-43fde7d8d80c"). InnerVolumeSpecName "kube-api-access-8xrq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:21.615234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.615200    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xrq6\" (UniqueName: \"kubernetes.io/projected/37746100-2141-412c-b8d3-43fde7d8d80c-kube-api-access-8xrq6\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.615234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:21.615235    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37746100-2141-412c-b8d3-43fde7d8d80c-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:22.026884 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.026849    5318 generic.go:356] "Generic (PLEG): container finished" podID="37746100-2141-412c-b8d3-43fde7d8d80c" containerID="85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4" exitCode=0
Nov 03 21:05:22.027071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.026891    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"
Nov 03 21:05:22.027071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.026932    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg" event={"ID":"37746100-2141-412c-b8d3-43fde7d8d80c","Type":"ContainerDied","Data":"85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4"}
Nov 03 21:05:22.027071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.026985    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg" event={"ID":"37746100-2141-412c-b8d3-43fde7d8d80c","Type":"ContainerDied","Data":"1d3c4e2de1b46cd240328ce32e82d2d0c7f383276afb0af0fc7a1a9ff5ebebb6"}
Nov 03 21:05:22.027071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.027007    5318 scope.go:122] "RemoveContainer" containerID="85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4"
Nov 03 21:05:22.037643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.037623    5318 scope.go:122] "RemoveContainer" containerID="85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4"
Nov 03 21:05:22.037932 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:05:22.037911    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4\": container with ID starting with 85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4 not found: ID does not exist" containerID="85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4"
Nov 03 21:05:22.037979 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.037944    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4"} err="failed to get container status \"85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4\": rpc error: code = NotFound desc = could not find container \"85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4\": container with ID starting with 85a2119c0b41596b85b0be149eeb9ba3b90304d02eb6c77a5ab35796b35e89b4 not found: ID does not exist"
Nov 03 21:05:22.182702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.182663    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"]
Nov 03 21:05:22.227132 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.227097    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-878695-55bd6dbdb7-ncwrg"]
Nov 03 21:05:22.451122 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:22.451091    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:23.264315 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:23.264280    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37746100-2141-412c-b8d3-43fde7d8d80c" path="/var/lib/kubelet/pods/37746100-2141-412c-b8d3-43fde7d8d80c/volumes"
Nov 03 21:05:24.265587 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:24.265547    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:05:25.155667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:25.155639    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:25.933904 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:05:25.933866    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:05:27.482630 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:27.482597    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:28.000829 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:28.000800    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:05:29.008000 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:29.007972    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:05:31.894289 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:31.894246    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:32.301454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:32.301352    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:35.519514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:35.519479    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:35.721953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:35.721920    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:36.236777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.236749    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"]
Nov 03 21:05:36.237005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.236945    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" podUID="cdb91df4-f000-4c39-80cb-87c04c11182b" containerName="maas-api" containerID="cri-o://5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e" gracePeriod=30
Nov 03 21:05:36.392226 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.392173    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:36.525726 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.525697    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 21:05:36.599399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.599361    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm\" (UniqueName: \"kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm\") pod \"cdb91df4-f000-4c39-80cb-87c04c11182b\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") "
Nov 03 21:05:36.599604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.599497    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs\") pod \"cdb91df4-f000-4c39-80cb-87c04c11182b\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") "
Nov 03 21:05:36.599604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.599542    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\") pod \"cdb91df4-f000-4c39-80cb-87c04c11182b\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") "
Nov 03 21:05:36.599604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.599568    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca\") pod \"cdb91df4-f000-4c39-80cb-87c04c11182b\" (UID: \"cdb91df4-f000-4c39-80cb-87c04c11182b\") "
Nov 03 21:05:36.601120 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.600850    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca" pod "cdb91df4-f000-4c39-80cb-87c04c11182b" (UID: "cdb91df4-f000-4c39-80cb-87c04c11182b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:36.603613 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.603579    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs" pod "cdb91df4-f000-4c39-80cb-87c04c11182b" (UID: "cdb91df4-f000-4c39-80cb-87c04c11182b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:36.603777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.603751    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls" pod "cdb91df4-f000-4c39-80cb-87c04c11182b" (UID: "cdb91df4-f000-4c39-80cb-87c04c11182b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:36.609396 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.609368    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm" pod "cdb91df4-f000-4c39-80cb-87c04c11182b" (UID: "cdb91df4-f000-4c39-80cb-87c04c11182b"). InnerVolumeSpecName "kube-api-access-ltgcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:36.701195 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.701160    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.701195 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.701192    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdb91df4-f000-4c39-80cb-87c04c11182b-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.701195 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.701202    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdb91df4-f000-4c39-80cb-87c04c11182b-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.701531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:36.701211    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltgcm\" (UniqueName: \"kubernetes.io/projected/cdb91df4-f000-4c39-80cb-87c04c11182b-kube-api-access-ltgcm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:37.126560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.126528    5318 generic.go:356] "Generic (PLEG): container finished" podID="cdb91df4-f000-4c39-80cb-87c04c11182b" containerID="5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e" exitCode=0
Nov 03 21:05:37.126747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.126579    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"
Nov 03 21:05:37.126747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.126609    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" event={"ID":"cdb91df4-f000-4c39-80cb-87c04c11182b","Type":"ContainerDied","Data":"5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e"}
Nov 03 21:05:37.126747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.126658    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg" event={"ID":"cdb91df4-f000-4c39-80cb-87c04c11182b","Type":"ContainerDied","Data":"15a8ed5acdb2d0417d1ba53395b9c81f570639e5b85c7dcd39ad80ef3f7ea468"}
Nov 03 21:05:37.126747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.126680    5318 scope.go:122] "RemoveContainer" containerID="5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e"
Nov 03 21:05:37.137138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.137119    5318 scope.go:122] "RemoveContainer" containerID="5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e"
Nov 03 21:05:37.137395 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:05:37.137378    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e\": container with ID starting with 5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e not found: ID does not exist" containerID="5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e"
Nov 03 21:05:37.137474 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.137401    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e"} err="failed to get container status \"5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e\": rpc error: code = NotFound desc = could not find container \"5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e\": container with ID starting with 5839cb4b0b03af396bf34caa0310265b81008bb3c8cbecb14c5029545de2873e not found: ID does not exist"
Nov 03 21:05:37.280925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.280897    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"]
Nov 03 21:05:37.287069 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.287038    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e427e7-559999dbfb-xl2vg"]
Nov 03 21:05:37.579958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.579869    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:37.871233 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:37.871200    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.124500 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.124423    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.269271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269237    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"]
Nov 03 21:05:38.269814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269798    5318 cpu_manager.go:414] "containerMap: removing container" podUID="37746100-2141-412c-b8d3-43fde7d8d80c" containerName="payload-pre-processing"
Nov 03 21:05:38.269814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269813    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37746100-2141-412c-b8d3-43fde7d8d80c" containerName="payload-pre-processing"
Nov 03 21:05:38.269955 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269846    5318 cpu_manager.go:414] "containerMap: removing container" podUID="cdb91df4-f000-4c39-80cb-87c04c11182b" containerName="maas-api"
Nov 03 21:05:38.269955 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269851    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb91df4-f000-4c39-80cb-87c04c11182b" containerName="maas-api"
Nov 03 21:05:38.269955 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269947    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb91df4-f000-4c39-80cb-87c04c11182b" containerName="maas-api"
Nov 03 21:05:38.270108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.269959    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="37746100-2141-412c-b8d3-43fde7d8d80c" containerName="payload-pre-processing"
Nov 03 21:05:38.274744 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.274721    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.326522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.326500    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e68f8112-dockercfg-bqnrr\""
Nov 03 21:05:38.326632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.326575    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e68f8112\""
Nov 03 21:05:38.376206 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.376128    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"]
Nov 03 21:05:38.410788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.410476    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"]
Nov 03 21:05:38.433378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.431278    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.439289 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.439251    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.439691 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.439666    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpk22\" (UniqueName: \"kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.455730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.455707    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"]
Nov 03 21:05:38.519227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.519199    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.540885 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.540859    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ktt\" (UniqueName: \"kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.541020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.540999    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.541170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.541149    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.541237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.541203    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpk22\" (UniqueName: \"kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.541905 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.541887    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.599052 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.599007    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpk22\" (UniqueName: \"kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.642089 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.642034    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ktt\" (UniqueName: \"kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.642197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.642102    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.642746 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.642729    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.701145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.701115    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ktt\" (UniqueName: \"kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.753370 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.753346    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:38.816194 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.816052    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:38.888726 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.888697    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:38.987277 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.987193    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"]
Nov 03 21:05:38.993259 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:38.993230    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e427e7-tl2fw"]
Nov 03 21:05:39.134116 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.134087    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"]
Nov 03 21:05:39.135484 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:05:39.135445    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545898ed_960d_4a1f_9dd9_44fe5d666f78.slice/crio-537a4f62a6720a9f4e0cdd7b5025e3ea2d7f9f031dce63fe1abfc6bc79139412 WatchSource:0}: Error finding container 537a4f62a6720a9f4e0cdd7b5025e3ea2d7f9f031dce63fe1abfc6bc79139412: Status 404 returned error can't find the container with id 537a4f62a6720a9f4e0cdd7b5025e3ea2d7f9f031dce63fe1abfc6bc79139412
Nov 03 21:05:39.256170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.256145    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20976a1d-9c35-4140-b1de-2a1a4da3175f" path="/var/lib/kubelet/pods/20976a1d-9c35-4140-b1de-2a1a4da3175f/volumes"
Nov 03 21:05:39.256615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.256599    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb91df4-f000-4c39-80cb-87c04c11182b" path="/var/lib/kubelet/pods/cdb91df4-f000-4c39-80cb-87c04c11182b/volumes"
Nov 03 21:05:39.262514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.262494    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"]
Nov 03 21:05:39.263542 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:05:39.263521    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded70b30c_5558_427b_bd32_d78ffb7a1a8e.slice/crio-d3a7c685eaf95e2cab513ad1d2c9392b6d443e3d46516771f8c92706dc487c8b WatchSource:0}: Error finding container d3a7c685eaf95e2cab513ad1d2c9392b6d443e3d46516771f8c92706dc487c8b: Status 404 returned error can't find the container with id d3a7c685eaf95e2cab513ad1d2c9392b6d443e3d46516771f8c92706dc487c8b
Nov 03 21:05:39.452794 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.452711    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.667010 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.666964    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.883469 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.883434    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:39.951601 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.951570    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"]
Nov 03 21:05:39.954081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:39.954060    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-2e3ea2-rmvgd"]
Nov 03 21:05:40.148947 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.148859    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw" event={"ID":"545898ed-960d-4a1f-9dd9-44fe5d666f78","Type":"ContainerStarted","Data":"942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d"}
Nov 03 21:05:40.148947 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.148915    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw" event={"ID":"545898ed-960d-4a1f-9dd9-44fe5d666f78","Type":"ContainerStarted","Data":"537a4f62a6720a9f4e0cdd7b5025e3ea2d7f9f031dce63fe1abfc6bc79139412"}
Nov 03 21:05:40.149176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.148954    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:40.150756 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.150729    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586" event={"ID":"ed70b30c-5558-427b-bd32-d78ffb7a1a8e","Type":"ContainerStarted","Data":"69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8"}
Nov 03 21:05:40.150877 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.150778    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586" event={"ID":"ed70b30c-5558-427b-bd32-d78ffb7a1a8e","Type":"ContainerStarted","Data":"d3a7c685eaf95e2cab513ad1d2c9392b6d443e3d46516771f8c92706dc487c8b"}
Nov 03 21:05:40.150877 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.150804    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:40.259829 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.259778    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw" podStartSLOduration=2.259765506 podStartE2EDuration="2.259765506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:40.257248206 +0000 UTC m=+2133.236805045" watchObservedRunningTime="2026-11-03 21:05:40.259765506 +0000 UTC m=+2133.239322345"
Nov 03 21:05:40.367576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.367533    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586" podStartSLOduration=2.367518967 podStartE2EDuration="2.367518967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:40.365201161 +0000 UTC m=+2133.344758002" watchObservedRunningTime="2026-11-03 21:05:40.367518967 +0000 UTC m=+2133.347075807"
Nov 03 21:05:40.456963 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:40.456890    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:41.255269 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:41.255240    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03630754-ba7b-4e10-94a8-9b91daa9b2bf" path="/var/lib/kubelet/pods/03630754-ba7b-4e10-94a8-9b91daa9b2bf/volumes"
Nov 03 21:05:43.714775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:43.714737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:43.925929 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:43.925891    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:44.632697 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:44.632663    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:46.156664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:46.156621    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:46.374787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:46.374752    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:46.587404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:46.587256    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:47.060247 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:47.060161    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:47.315979 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:47.315901    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:47.548050 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:47.548019    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:50.027654 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:50.027618    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:51.159761 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:51.159735    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:05:51.160122 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:51.159801    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:05:55.047246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:55.047215    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:55.933750 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:05:55.933714    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:05:59.013136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.013099    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"]
Nov 03 21:05:59.171132 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.171092    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"]
Nov 03 21:05:59.171347 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.171298    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" podUID="e66c23a8-cb01-4760-862d-735552c14236" containerName="maas-api" containerID="cri-o://b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb" gracePeriod=30
Nov 03 21:05:59.330062 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.330036    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:59.490896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.490650    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:05:59.527701 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.527669    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:59.614801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.613702    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca\") pod \"e66c23a8-cb01-4760-862d-735552c14236\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") "
Nov 03 21:05:59.614801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.613790    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\") pod \"e66c23a8-cb01-4760-862d-735552c14236\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") "
Nov 03 21:05:59.614801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.613860    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs\") pod \"e66c23a8-cb01-4760-862d-735552c14236\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") "
Nov 03 21:05:59.614801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.613938    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5\" (UniqueName: \"kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5\") pod \"e66c23a8-cb01-4760-862d-735552c14236\" (UID: \"e66c23a8-cb01-4760-862d-735552c14236\") "
Nov 03 21:05:59.619606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.619384    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls" pod "e66c23a8-cb01-4760-862d-735552c14236" (UID: "e66c23a8-cb01-4760-862d-735552c14236"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:59.619917 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.619882    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca" pod "e66c23a8-cb01-4760-862d-735552c14236" (UID: "e66c23a8-cb01-4760-862d-735552c14236"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:59.623470 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.623433    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5" pod "e66c23a8-cb01-4760-862d-735552c14236" (UID: "e66c23a8-cb01-4760-862d-735552c14236"). InnerVolumeSpecName "kube-api-access-455t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:59.625462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.625425    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs" pod "e66c23a8-cb01-4760-862d-735552c14236" (UID: "e66c23a8-cb01-4760-862d-735552c14236"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:59.715896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.715863    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66c23a8-cb01-4760-862d-735552c14236-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:59.715896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.715894    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:59.715896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.715904    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66c23a8-cb01-4760-862d-735552c14236-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:59.716207 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:05:59.715913    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455t5\" (UniqueName: \"kubernetes.io/projected/e66c23a8-cb01-4760-862d-735552c14236-kube-api-access-455t5\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:00.311673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.311637    5318 generic.go:356] "Generic (PLEG): container finished" podID="e66c23a8-cb01-4760-862d-735552c14236" containerID="b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb" exitCode=0
Nov 03 21:06:00.312274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.311698    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" event={"ID":"e66c23a8-cb01-4760-862d-735552c14236","Type":"ContainerDied","Data":"b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb"}
Nov 03 21:06:00.312274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.311706    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"
Nov 03 21:06:00.312274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.311743    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88" event={"ID":"e66c23a8-cb01-4760-862d-735552c14236","Type":"ContainerDied","Data":"80ee604c6e0fecdc43cee78d36227d0664142f19c685b123e68600e645f84cba"}
Nov 03 21:06:00.312274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.311765    5318 scope.go:122] "RemoveContainer" containerID="b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb"
Nov 03 21:06:00.322383 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.322363    5318 scope.go:122] "RemoveContainer" containerID="b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb"
Nov 03 21:06:00.322671 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:06:00.322653    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb\": container with ID starting with b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb not found: ID does not exist" containerID="b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb"
Nov 03 21:06:00.322734 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.322679    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb"} err="failed to get container status \"b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb\": rpc error: code = NotFound desc = could not find container \"b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb\": container with ID starting with b5ea76b583fbfa96dbdebd223fb88fed06fb6f91c9560f32e9d12700ef4d3eeb not found: ID does not exist"
Nov 03 21:06:00.453449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.453414    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:00.466516 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.466481    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"]
Nov 03 21:06:00.513972 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.513939    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-37c5b0-598994d9f9-hxq88"]
Nov 03 21:06:00.806346 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:00.806274    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:01.076588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:01.076515    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:01.255930 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:01.255895    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66c23a8-cb01-4760-862d-735552c14236" path="/var/lib/kubelet/pods/e66c23a8-cb01-4760-862d-735552c14236/volumes"
Nov 03 21:06:01.347004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:01.346972    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:01.566861 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:01.566829    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:01.964106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:01.964074    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:02.170494 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:02.170456    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:04.015041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.014994    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8" podUID="7d79e7f1-10da-4071-b011-dcdbc254136a" containerName="payload-pre-processing" containerID="cri-o://71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9" gracePeriod=25
Nov 03 21:06:04.284532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.284472    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:06:04.340689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.340655    5318 generic.go:356] "Generic (PLEG): container finished" podID="7d79e7f1-10da-4071-b011-dcdbc254136a" containerID="71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9" exitCode=0
Nov 03 21:06:04.340864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.340708    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"
Nov 03 21:06:04.340864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.340755    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8" event={"ID":"7d79e7f1-10da-4071-b011-dcdbc254136a","Type":"ContainerDied","Data":"71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9"}
Nov 03 21:06:04.340864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.340826    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8" event={"ID":"7d79e7f1-10da-4071-b011-dcdbc254136a","Type":"ContainerDied","Data":"72477f225c9432fae7ec02ab8e2738f31ac4436cb89712b331fee479c11cf15f"}
Nov 03 21:06:04.340864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.340859    5318 scope.go:122] "RemoveContainer" containerID="71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9"
Nov 03 21:06:04.353119 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.353101    5318 scope.go:122] "RemoveContainer" containerID="71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9"
Nov 03 21:06:04.353419 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:06:04.353396    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9\": container with ID starting with 71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9 not found: ID does not exist" containerID="71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9"
Nov 03 21:06:04.353484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.353430    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9"} err="failed to get container status \"71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9\": rpc error: code = NotFound desc = could not find container \"71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9\": container with ID starting with 71942d33e75616ddf900dca1be34afabeda2adb701f60220cdd1c34426fd33f9 not found: ID does not exist"
Nov 03 21:06:04.387271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.387240    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q\" (UniqueName: \"kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q\") pod \"7d79e7f1-10da-4071-b011-dcdbc254136a\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") "
Nov 03 21:06:04.387513 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.387494    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume\") pod \"7d79e7f1-10da-4071-b011-dcdbc254136a\" (UID: \"7d79e7f1-10da-4071-b011-dcdbc254136a\") "
Nov 03 21:06:04.387970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.387944    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume" pod "7d79e7f1-10da-4071-b011-dcdbc254136a" (UID: "7d79e7f1-10da-4071-b011-dcdbc254136a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:04.389418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.389399    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q" pod "7d79e7f1-10da-4071-b011-dcdbc254136a" (UID: "7d79e7f1-10da-4071-b011-dcdbc254136a"). InnerVolumeSpecName "kube-api-access-6mf7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:04.489755 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.489719    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d79e7f1-10da-4071-b011-dcdbc254136a-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:04.489755 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.489752    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mf7q\" (UniqueName: \"kubernetes.io/projected/7d79e7f1-10da-4071-b011-dcdbc254136a-kube-api-access-6mf7q\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:04.760102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.760067    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"]
Nov 03 21:06:04.767887 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:04.767854    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8448c9-8666b7848c-mvrx8"]
Nov 03 21:06:05.272360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:05.266053    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d79e7f1-10da-4071-b011-dcdbc254136a" path="/var/lib/kubelet/pods/7d79e7f1-10da-4071-b011-dcdbc254136a/volumes"
Nov 03 21:06:07.051236 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:07.051206    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:09.084813 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:09.084784    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:09.303139 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:09.303092    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:09.555188 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:09.555146    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:10.550689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:10.550656    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:10.780650 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:10.780624    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:11.827153 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:11.827113    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:12.052305 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:12.052273    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:12.356448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:12.356416    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:12.615519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:12.615446    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:17.920397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:17.920313    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:23.030241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:23.030198    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:25.478538 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:25.478496    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:25.934180 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:06:25.934146    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:06:27.865786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:27.865712    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:28.078113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:28.078082    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:29.917027 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:29.916987    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:06:30.476055 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:30.476022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:31.673331 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:31.673270    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:31.887866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:31.887829    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:32.386770 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:32.386735    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:33.400885 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.400852    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"]
Nov 03 21:06:33.401784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.401750    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94" podUID="56ffbe61-2fdb-4173-ba5d-2c3f1b972545" containerName="maas-api" containerID="cri-o://98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21" gracePeriod=30
Nov 03 21:06:33.568597 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.568535    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:33.917249 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.917214    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 21:06:33.941850 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.941809    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:33.961603 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.961530    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh\" (UniqueName: \"kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh\") pod \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") "
Nov 03 21:06:33.961777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.961601    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs\") pod \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") "
Nov 03 21:06:33.961777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.961656    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\") pod \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") "
Nov 03 21:06:33.961777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.961744    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca\") pod \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\" (UID: \"56ffbe61-2fdb-4173-ba5d-2c3f1b972545\") "
Nov 03 21:06:33.962404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.962367    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca" pod "56ffbe61-2fdb-4173-ba5d-2c3f1b972545" (UID: "56ffbe61-2fdb-4173-ba5d-2c3f1b972545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:33.964195 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.964169    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs" pod "56ffbe61-2fdb-4173-ba5d-2c3f1b972545" (UID: "56ffbe61-2fdb-4173-ba5d-2c3f1b972545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:33.964388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.964370    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls" pod "56ffbe61-2fdb-4173-ba5d-2c3f1b972545" (UID: "56ffbe61-2fdb-4173-ba5d-2c3f1b972545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:33.964537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:33.964521    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh" pod "56ffbe61-2fdb-4173-ba5d-2c3f1b972545" (UID: "56ffbe61-2fdb-4173-ba5d-2c3f1b972545"). InnerVolumeSpecName "kube-api-access-ww6wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:34.063378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.063243    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6wh\" (UniqueName: \"kubernetes.io/projected/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-kube-api-access-ww6wh\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:34.063378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.063277    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:34.063378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.063293    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:34.063378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.063306    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ffbe61-2fdb-4173-ba5d-2c3f1b972545-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:34.296969 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.296886    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:34.569519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.569445    5318 generic.go:356] "Generic (PLEG): container finished" podID="56ffbe61-2fdb-4173-ba5d-2c3f1b972545" containerID="98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21" exitCode=0
Nov 03 21:06:34.569519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.569489    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"
Nov 03 21:06:34.569996 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.569523    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94" event={"ID":"56ffbe61-2fdb-4173-ba5d-2c3f1b972545","Type":"ContainerDied","Data":"98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21"}
Nov 03 21:06:34.569996 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.569576    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94" event={"ID":"56ffbe61-2fdb-4173-ba5d-2c3f1b972545","Type":"ContainerDied","Data":"1eabd82995c3d6189244e28fa904776a8fe6288d2e7dc0cd74865e6418062b6d"}
Nov 03 21:06:34.569996 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.569599    5318 scope.go:122] "RemoveContainer" containerID="98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21"
Nov 03 21:06:34.580024 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.580009    5318 scope.go:122] "RemoveContainer" containerID="98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21"
Nov 03 21:06:34.580307 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:06:34.580290    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21\": container with ID starting with 98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21 not found: ID does not exist" containerID="98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21"
Nov 03 21:06:34.580387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.580316    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21"} err="failed to get container status \"98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21\": rpc error: code = NotFound desc = could not find container \"98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21\": container with ID starting with 98d5fcb47282de54fe05e27133f08ba78ff0b44e579aeda960659775d91f1d21 not found: ID does not exist"
Nov 03 21:06:34.724266 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.724238    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"]
Nov 03 21:06:34.770170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:34.770146    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba7e83-6868888869-ntq94"]
Nov 03 21:06:35.240774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:35.240744    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:35.258949 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:35.258915    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ffbe61-2fdb-4173-ba5d-2c3f1b972545" path="/var/lib/kubelet/pods/56ffbe61-2fdb-4173-ba5d-2c3f1b972545/volumes"
Nov 03 21:06:35.459470 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:35.459420    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:35.668703 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:35.668670    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:35.896191 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:35.896143    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:36.113802 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:36.113772    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:36.477639 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:36.477553    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:37.163492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:37.163464    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:40.328855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:40.328825    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:40.529784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:40.529751    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:40.903156 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:40.903117    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:42.355907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:42.355873    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:42.586548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:42.586473    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:42.813199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:42.813032    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:43.573380 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573325    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"]
Nov 03 21:06:43.573908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573894    5318 cpu_manager.go:414] "containerMap: removing container" podUID="7d79e7f1-10da-4071-b011-dcdbc254136a" containerName="payload-pre-processing"
Nov 03 21:06:43.573954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573908    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d79e7f1-10da-4071-b011-dcdbc254136a" containerName="payload-pre-processing"
Nov 03 21:06:43.573954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573926    5318 cpu_manager.go:414] "containerMap: removing container" podUID="e66c23a8-cb01-4760-862d-735552c14236" containerName="maas-api"
Nov 03 21:06:43.573954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573932    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66c23a8-cb01-4760-862d-735552c14236" containerName="maas-api"
Nov 03 21:06:43.574045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573971    5318 cpu_manager.go:414] "containerMap: removing container" podUID="56ffbe61-2fdb-4173-ba5d-2c3f1b972545" containerName="maas-api"
Nov 03 21:06:43.574045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.573976    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ffbe61-2fdb-4173-ba5d-2c3f1b972545" containerName="maas-api"
Nov 03 21:06:43.574109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.574064    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d79e7f1-10da-4071-b011-dcdbc254136a" containerName="payload-pre-processing"
Nov 03 21:06:43.574109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.574073    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66c23a8-cb01-4760-862d-735552c14236" containerName="maas-api"
Nov 03 21:06:43.574109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.574084    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ffbe61-2fdb-4173-ba5d-2c3f1b972545" containerName="maas-api"
Nov 03 21:06:43.579480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.579459    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.627939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.627913    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:06:43.674421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.674380    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:06:43.675234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.675208    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxtl\" (UniqueName: \"kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.675331 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.675307    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.675572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.675548    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.677542 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.677521    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"]
Nov 03 21:06:43.776709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.776676    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.776852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.776760    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxtl\" (UniqueName: \"kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.776903 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.776847    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.777470 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.777446    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.778993 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.778973    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.834508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.834451    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxtl\" (UniqueName: \"kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl\") pod \"maas-api-revoke-keys-e2e-api-b-0776c6-frhq8\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:43.895194 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:43.895170    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:44.263808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:44.263783    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"]
Nov 03 21:06:44.266549 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:06:44.266518    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f6fc77_3c95_4e0a_93fd_99d867f19ce7.slice/crio-67df33d18f8d2d25a2ac167a83cb0e6daa4cb809729d798440c0c8a5d754538d WatchSource:0}: Error finding container 67df33d18f8d2d25a2ac167a83cb0e6daa4cb809729d798440c0c8a5d754538d: Status 404 returned error can't find the container with id 67df33d18f8d2d25a2ac167a83cb0e6daa4cb809729d798440c0c8a5d754538d
Nov 03 21:06:44.642275 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:44.642240    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8" event={"ID":"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7","Type":"ContainerStarted","Data":"d3a551ed1ccd8021a66131dabd091a7272e3f9e41ca80413e37f6bb7979c94a7"}
Nov 03 21:06:44.642617 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:44.642303    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8" event={"ID":"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7","Type":"ContainerStarted","Data":"67df33d18f8d2d25a2ac167a83cb0e6daa4cb809729d798440c0c8a5d754538d"}
Nov 03 21:06:44.751242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:44.751198    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8" podStartSLOduration=1.751183698 podStartE2EDuration="1.751183698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:44.749975389 +0000 UTC m=+2197.729532228" watchObservedRunningTime="2026-11-03 21:06:44.751183698 +0000 UTC m=+2197.730740559"
Nov 03 21:06:44.943830 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:44.943797    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:45.649360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:45.649313    5318 generic.go:356] "Generic (PLEG): container finished" podID="a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" containerID="d3a551ed1ccd8021a66131dabd091a7272e3f9e41ca80413e37f6bb7979c94a7" exitCode=0
Nov 03 21:06:45.649728 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:45.649367    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8" event={"ID":"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7","Type":"ContainerDied","Data":"d3a551ed1ccd8021a66131dabd091a7272e3f9e41ca80413e37f6bb7979c94a7"}
Nov 03 21:06:46.332106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.332076    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:46.808332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.808308    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:46.914902 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.914827    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca\") pod \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") "
Nov 03 21:06:46.915032 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.914949    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl\" (UniqueName: \"kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl\") pod \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") "
Nov 03 21:06:46.915170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.915137    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca\") pod \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\" (UID: \"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7\") "
Nov 03 21:06:46.915568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.915539    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca" pod "a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" (UID: "a7f6fc77-3c95-4e0a-93fd-99d867f19ce7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:46.917117 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.917087    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca" pod "a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" (UID: "a7f6fc77-3c95-4e0a-93fd-99d867f19ce7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:46.917196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:46.917110    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl" pod "a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" (UID: "a7f6fc77-3c95-4e0a-93fd-99d867f19ce7"). InnerVolumeSpecName "kube-api-access-2lxtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:47.016867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:47.016835    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:47.016867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:47.016861    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:47.016867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:47.016871    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxtl\" (UniqueName: \"kubernetes.io/projected/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7-kube-api-access-2lxtl\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:47.663199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:47.663163    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8" event={"ID":"a7f6fc77-3c95-4e0a-93fd-99d867f19ce7","Type":"ContainerDied","Data":"67df33d18f8d2d25a2ac167a83cb0e6daa4cb809729d798440c0c8a5d754538d"}
Nov 03 21:06:47.663199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:47.663196    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"
Nov 03 21:06:47.663438 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:47.663214    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67df33d18f8d2d25a2ac167a83cb0e6daa4cb809729d798440c0c8a5d754538d"
Nov 03 21:06:50.777925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:50.777893    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:51.007857 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:51.007815    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:51.218593 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:51.218539    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:51.600131 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:51.600095    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:52.052167 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.052088    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"]
Nov 03 21:06:52.052710 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.052692    5318 cpu_manager.go:414] "containerMap: removing container" podUID="a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" containerName="revoke-keys"
Nov 03 21:06:52.052763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.052712    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" containerName="revoke-keys"
Nov 03 21:06:52.052814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.052805    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" containerName="revoke-keys"
Nov 03 21:06:52.056457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.056433    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.180667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.180631    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"]
Nov 03 21:06:52.184282 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184255    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184459 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184442    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184541 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184476    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184563    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184632    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184715 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184676    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8dt\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184773 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184709    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184833 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184776    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.184955 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.184836    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.285939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.285904    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.285996    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286178 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286148    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286220 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286190    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286237    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286276    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286313    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8dt\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286505 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286361    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286505 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286425    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.286505 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286431    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.288134 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.286653    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.288134 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.287316    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.288134 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.287363    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.288134 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.288006    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.288517 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.288489    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.289540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.289471    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.349283 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.349236    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.349744 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.349721    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8dt\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.372559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.372534    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:52.800740 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.800645    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"]
Nov 03 21:06:52.804912 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:06:52.804884    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1e7eff_b179_453a_ac54_1197a1b45f0f.slice/crio-b1d313713feabb33bd7c70a35e81afec0a5ac62c39247d109abea36000cba420 WatchSource:0}: Error finding container b1d313713feabb33bd7c70a35e81afec0a5ac62c39247d109abea36000cba420: Status 404 returned error can't find the container with id b1d313713feabb33bd7c70a35e81afec0a5ac62c39247d109abea36000cba420
Nov 03 21:06:52.808137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.808100    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:06:52.808214 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.808187    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:06:52.808257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:52.808232    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:06:53.463017 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.462976    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"]
Nov 03 21:06:53.482837 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.482804    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.527304 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.527268    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"]
Nov 03 21:06:53.607995 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.606629    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj962\" (UniqueName: \"kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.607995 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.606759    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.607995 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.606851    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.609015 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.608984    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:06:53.609224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.609190    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:06:53.705199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.705164    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" event={"ID":"fb1e7eff-b179-453a-ac54-1197a1b45f0f","Type":"ContainerStarted","Data":"bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b"}
Nov 03 21:06:53.705393 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.705225    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" event={"ID":"fb1e7eff-b179-453a-ac54-1197a1b45f0f","Type":"ContainerStarted","Data":"b1d313713feabb33bd7c70a35e81afec0a5ac62c39247d109abea36000cba420"}
Nov 03 21:06:53.708037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.708008    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj962\" (UniqueName: \"kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.708163 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.708065    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.708163 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.708126    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.708767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.708746    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.710697 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.710669    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.781407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.772692    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj962\" (UniqueName: \"kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.818190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.818161    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:53.822072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.822032    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podStartSLOduration=1.822018232 podStartE2EDuration="1.822018232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:53.820034318 +0000 UTC m=+2206.799591158" watchObservedRunningTime="2026-11-03 21:06:53.822018232 +0000 UTC m=+2206.801575072"
Nov 03 21:06:53.918550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:53.917672    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:54.229086 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.229060    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"]
Nov 03 21:06:54.231401 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:06:54.231376    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7fab6a_f141_4651_9376_3b80b3fe93c5.slice/crio-e0b7533f99cec7c40b5f2e98f16dddac4298ad8f931214e87a9b9f665b914d64 WatchSource:0}: Error finding container e0b7533f99cec7c40b5f2e98f16dddac4298ad8f931214e87a9b9f665b914d64: Status 404 returned error can't find the container with id e0b7533f99cec7c40b5f2e98f16dddac4298ad8f931214e87a9b9f665b914d64
Nov 03 21:06:54.379367 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.378259    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:06:54.379548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.379409    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:06:54.379548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.379458    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:06:54.713482 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.713444    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s" event={"ID":"9f7fab6a-f141-4651-9376-3b80b3fe93c5","Type":"ContainerStarted","Data":"8549554a7867a4ebc523faeac30bb7f4e1c216b3fa47d8d23485d60169a6311f"}
Nov 03 21:06:54.713861 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.713498    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s" event={"ID":"9f7fab6a-f141-4651-9376-3b80b3fe93c5","Type":"ContainerStarted","Data":"e0b7533f99cec7c40b5f2e98f16dddac4298ad8f931214e87a9b9f665b914d64"}
Nov 03 21:06:54.825548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:54.825455    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s" podStartSLOduration=1.825437838 podStartE2EDuration="1.825437838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:54.823131495 +0000 UTC m=+2207.802688336" watchObservedRunningTime="2026-11-03 21:06:54.825437838 +0000 UTC m=+2207.804994679"
Nov 03 21:06:55.121592 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:55.121561    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:55.362553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:55.362527    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:55.374104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:55.373643    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:06:55.374104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:55.373709    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:06:55.722242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:55.722155    5318 generic.go:356] "Generic (PLEG): container finished" podID="9f7fab6a-f141-4651-9376-3b80b3fe93c5" containerID="8549554a7867a4ebc523faeac30bb7f4e1c216b3fa47d8d23485d60169a6311f" exitCode=0
Nov 03 21:06:55.722649 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:55.722238    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s" event={"ID":"9f7fab6a-f141-4651-9376-3b80b3fe93c5","Type":"ContainerDied","Data":"8549554a7867a4ebc523faeac30bb7f4e1c216b3fa47d8d23485d60169a6311f"}
Nov 03 21:06:55.946297 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:06:55.946254    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:06:56.378778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.378742    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:06:56.379057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.379028    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:06:56.929669 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.929645    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:56.952030 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.952008    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962\" (UniqueName: \"kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962\") pod \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") "
Nov 03 21:06:56.952158 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.952053    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca\") pod \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") "
Nov 03 21:06:56.952274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.952191    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca\") pod \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\" (UID: \"9f7fab6a-f141-4651-9376-3b80b3fe93c5\") "
Nov 03 21:06:56.952462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.952437    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca" pod "9f7fab6a-f141-4651-9376-3b80b3fe93c5" (UID: "9f7fab6a-f141-4651-9376-3b80b3fe93c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:56.952691 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.952669    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:56.954368 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.954320    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca" pod "9f7fab6a-f141-4651-9376-3b80b3fe93c5" (UID: "9f7fab6a-f141-4651-9376-3b80b3fe93c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:56.954453 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:56.954357    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962" pod "9f7fab6a-f141-4651-9376-3b80b3fe93c5" (UID: "9f7fab6a-f141-4651-9376-3b80b3fe93c5"). InnerVolumeSpecName "kube-api-access-qj962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:57.054145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.054041    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f7fab6a-f141-4651-9376-3b80b3fe93c5-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:57.054145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.054072    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj962\" (UniqueName: \"kubernetes.io/projected/9f7fab6a-f141-4651-9376-3b80b3fe93c5-kube-api-access-qj962\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:57.374693 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.374659    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:06:57.396562 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.396115    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:06:57.516400 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.515514    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:57.739237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.739158    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"
Nov 03 21:06:57.739586 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.739174    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s" event={"ID":"9f7fab6a-f141-4651-9376-3b80b3fe93c5","Type":"ContainerDied","Data":"e0b7533f99cec7c40b5f2e98f16dddac4298ad8f931214e87a9b9f665b914d64"}
Nov 03 21:06:57.739699 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:57.739641    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b7533f99cec7c40b5f2e98f16dddac4298ad8f931214e87a9b9f665b914d64"
Nov 03 21:06:58.378905 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:58.378433    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:06:58.378905 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:58.378493    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:06:59.373454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:59.373415    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:06:59.373632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:06:59.373483    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:07:00.373411 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:00.373378    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 21:07:00.373958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:00.373429    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 21:07:01.449199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:01.449164    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:07:01.449678 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:01.449544    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:07:01.450407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:01.450389    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:07:02.289944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:02.289898    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:04.387513 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:04.387482    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:04.597584 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:04.597545    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:04.824185 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:04.824074    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:05.477608 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:05.477570    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:06.340377 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:06.340314    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:06.551152 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:06.551118    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:09.854354 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:09.854309    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:11.660170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:11.660134    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:12.009250 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:12.009018    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:12.231296 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:12.231261    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:14.838002 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:14.837958    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:16.542473 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:16.542439    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:16.832214 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:16.832129    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:17.034155 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:17.034127    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:22.024198 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:22.024162    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:22.975572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:22.975539    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:23.613258 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:23.613220    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:23.948245 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:23.948150    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:24.167216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:24.167174    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:24.733489 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:24.733456    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:25.604224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.604195    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"]
Nov 03 21:07:25.604774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.604761    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9f7fab6a-f141-4651-9376-3b80b3fe93c5" containerName="revoke-keys"
Nov 03 21:07:25.604829 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.604776    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7fab6a-f141-4651-9376-3b80b3fe93c5" containerName="revoke-keys"
Nov 03 21:07:25.604897 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.604888    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7fab6a-f141-4651-9376-3b80b3fe93c5" containerName="revoke-keys"
Nov 03 21:07:25.609843 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.609825    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.658471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.658450    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e68f8112-openshift-default-dockercfg-cp47h\""
Nov 03 21:07:25.726016 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.725983    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726061    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726150    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726180    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726208    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726231    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726255    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvzf\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726316    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.726373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.726366    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.804197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.804160    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:25.814485 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.814458    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"]
Nov 03 21:07:25.828090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828057    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828240 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828171    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828240 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828210    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828243    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828269    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828294    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvzf\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828369    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828408    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828541    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.828673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.828608    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.829203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.829178    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.829473 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.829451    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.829889 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.829864    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.830208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.830186    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.831398 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.831319    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.832310 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.832287    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.886470 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.886416    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvzf\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.886578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.886561    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.923655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:25.923620    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:25.939990 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:07:25.939960    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:07:26.358607 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:26.358559    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"]
Nov 03 21:07:26.359372 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:07:26.359326    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fcd3de_0656_4b21_a46e_3e76fd5fc6be.slice/crio-3fab6d8deb026e49f54213025de6d325ebb9b65b3acfcc880c8adc507ee4a575 WatchSource:0}: Error finding container 3fab6d8deb026e49f54213025de6d325ebb9b65b3acfcc880c8adc507ee4a575: Status 404 returned error can't find the container with id 3fab6d8deb026e49f54213025de6d325ebb9b65b3acfcc880c8adc507ee4a575
Nov 03 21:07:26.362044 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:26.362006    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:26.362129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:26.362074    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:26.362129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:26.362104    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:26.996607 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:26.996576    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" event={"ID":"40fcd3de-0656-4b21-a46e-3e76fd5fc6be","Type":"ContainerStarted","Data":"cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5"}
Nov 03 21:07:26.997005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:26.996629    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" event={"ID":"40fcd3de-0656-4b21-a46e-3e76fd5fc6be","Type":"ContainerStarted","Data":"3fab6d8deb026e49f54213025de6d325ebb9b65b3acfcc880c8adc507ee4a575"}
Nov 03 21:07:27.150678 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:27.150612    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podStartSLOduration=2.150591673 podStartE2EDuration="2.150591673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:27.150369219 +0000 UTC m=+2240.129926064" watchObservedRunningTime="2026-11-03 21:07:27.150591673 +0000 UTC m=+2240.130148514"
Nov 03 21:07:27.159783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:27.159749    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:27.476071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:27.476033    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:27.924803 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:27.924767    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:27.926113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:27.926086    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:27.926234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:27.926134    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 21:07:28.443472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:28.443441    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:28.701452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:28.689026    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:28.924806 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:28.924771    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:28.924970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:28.924833    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 21:07:29.924489 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:29.924450    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:29.924973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:29.924502    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 21:07:30.923987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:30.923949    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:30.924199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:30.924011    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 21:07:31.736176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:31.736126    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:31.924165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:31.924126    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 21:07:31.924387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:31.924183    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 21:07:31.938571 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:31.938544    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:32.802842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:32.802812    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:32.929277 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:32.929246    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:32.929612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:32.929584    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:32.930374 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:32.930325    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:07:34.289705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:34.289668    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:34.494384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:34.494356    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:34.703154 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:34.703123    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:35.490125 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:35.490087    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:36.106208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:36.106161    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:36.669434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:36.669386    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:38.343525 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:38.343483    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:40.718876 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.718841    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"]
Nov 03 21:07:40.722842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.722823    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.821191 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.821158    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"]
Nov 03 21:07:40.864885 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.864844    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.864957    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865007    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mqg\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865047    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865183    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865219    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865249    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865301    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.865452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.865360    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967003    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967291 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967114    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967291 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967165    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967291 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967208    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967291 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967238    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967291 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967281    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967591 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967394    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967591 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967447    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.967591 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.967499    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mqg\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.968640 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.968613    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.969400 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.968694    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.969566 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.968891    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.969681 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.969054    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.969784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.969284    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.973131 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.973108    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:40.973614 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:40.973593    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:41.024569 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.024542    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:41.024709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.024675    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mqg\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:41.037486 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.037464    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:41.503729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.503704    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"]
Nov 03 21:07:41.505856 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:07:41.505828    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e65ae6_cd43_447e_9665_af5734e5413e.slice/crio-72c30b9ad72bea659af14f0fc303acc685d107cc4a2f988cd4b5b0cded22481b WatchSource:0}: Error finding container 72c30b9ad72bea659af14f0fc303acc685d107cc4a2f988cd4b5b0cded22481b: Status 404 returned error can't find the container with id 72c30b9ad72bea659af14f0fc303acc685d107cc4a2f988cd4b5b0cded22481b
Nov 03 21:07:41.508402 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.508369    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:41.508508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.508435    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:41.508508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:41.508463    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:42.100204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:42.100167    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" event={"ID":"60e65ae6-cd43-447e-9665-af5734e5413e","Type":"ContainerStarted","Data":"011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8"}
Nov 03 21:07:42.100722 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:42.100221    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" event={"ID":"60e65ae6-cd43-447e-9665-af5734e5413e","Type":"ContainerStarted","Data":"72c30b9ad72bea659af14f0fc303acc685d107cc4a2f988cd4b5b0cded22481b"}
Nov 03 21:07:42.214110 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:42.214058    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podStartSLOduration=2.214043469 podStartE2EDuration="2.214043469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:42.212884364 +0000 UTC m=+2255.192441204" watchObservedRunningTime="2026-11-03 21:07:42.214043469 +0000 UTC m=+2255.193600303"
Nov 03 21:07:43.038576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:43.038542    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:43.040226 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:43.040201    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:07:43.040368 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:43.040248    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 21:07:43.821023 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:43.820966    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:44.038759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:44.038719    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:07:44.038963 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:44.038772    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 21:07:45.038071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:45.038036    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:07:45.038508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:45.038084    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 21:07:46.038010 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:46.037935    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:07:46.038010 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:46.037988    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 21:07:47.038747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:47.038706    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:07:47.039217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:47.038777    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 21:07:48.038563 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:48.038522    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 21:07:48.038719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:48.038612    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 21:07:49.043332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:49.043301    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:49.043777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:49.043712    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:49.044493 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:49.044474    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:07:49.158150 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:49.158094    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:50.449611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:50.449582    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:52.415251 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:52.415218    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:07:54.687117 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:54.687084    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:55.844386 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.844347    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"]
Nov 03 21:07:55.848904 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.848877    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.939899 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:07:55.939865    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:07:55.972895 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.972857    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.972981    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973154 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973127    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9n2q\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973189    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973263 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973249    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973327 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973306    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973429    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973534 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973509    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:55.973595 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:55.973570    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.049428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.049393    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"]
Nov 03 21:07:56.074702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.074670    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.074907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.074779    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9n2q\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.074907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.074835    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.074907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.074871    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.074907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.074903    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.075128 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.074978    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.075128 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.075009    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.075128 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.075059    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.075128 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.075120    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.075870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.075798    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.075870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.075819    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.076245 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.076222    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.076452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.076428    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.076545 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.076505    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.078289 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.078257    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.079058 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.079034    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.132895 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.132822    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.132895 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.132878    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9n2q\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.162827 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.162789    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:56.585621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.585590    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"]
Nov 03 21:07:56.587752 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:07:56.587722    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2776d738_536e_40aa_b7aa_8f96546f1758.slice/crio-315367e33fa3eced22db597856066484b4842c4a2e3480560421b727b882f50a WatchSource:0}: Error finding container 315367e33fa3eced22db597856066484b4842c4a2e3480560421b727b882f50a: Status 404 returned error can't find the container with id 315367e33fa3eced22db597856066484b4842c4a2e3480560421b727b882f50a
Nov 03 21:07:56.591084 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.591044    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:56.591157 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.591133    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:56.591192 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:56.591176    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:07:57.211553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:57.211513    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" event={"ID":"2776d738-536e-40aa-b7aa-8f96546f1758","Type":"ContainerStarted","Data":"55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a"}
Nov 03 21:07:57.211962 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:57.211570    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" event={"ID":"2776d738-536e-40aa-b7aa-8f96546f1758","Type":"ContainerStarted","Data":"315367e33fa3eced22db597856066484b4842c4a2e3480560421b727b882f50a"}
Nov 03 21:07:57.352164 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:57.352099    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podStartSLOduration=2.352079777 podStartE2EDuration="2.352079777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:57.34758803 +0000 UTC m=+2270.327144870" watchObservedRunningTime="2026-11-03 21:07:57.352079777 +0000 UTC m=+2270.331636619"
Nov 03 21:07:58.164047 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:58.164008    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:07:58.196525 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:58.196491    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:07:58.196700 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:58.196548    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 21:07:58.694361 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:58.693481    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:58.899746 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:58.899700    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:59.163532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:59.163500    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:07:59.163701 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:59.163555    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 21:07:59.561509 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:07:59.561358    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:00.163481 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:00.163450    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:08:00.163883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:00.163503    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 21:08:00.979359 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:00.975597    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:01.163661 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.163631    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:08:01.164035 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.163675    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 21:08:01.620946 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.620907    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6927-5ca86c"]
Nov 03 21:08:01.625628 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.625605    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:01.657539 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.657511    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k287\" (UniqueName: \"kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287\") pod \"test-curl-6927-5ca86c\" (UID: \"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d\") " pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:01.687359 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.686487    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6927-5ca86c"]
Nov 03 21:08:01.728183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.728152    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2jkh\""
Nov 03 21:08:01.758874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.758846    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k287\" (UniqueName: \"kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287\") pod \"test-curl-6927-5ca86c\" (UID: \"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d\") " pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:01.835854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.835824    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k287\" (UniqueName: \"kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287\") pod \"test-curl-6927-5ca86c\" (UID: \"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d\") " pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:01.924792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.924716    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:01.939653 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:01.939628    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:02.149251 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.149215    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:02.163797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.163771    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 21:08:02.164137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.163816    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 21:08:02.352501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.352471    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6927-5ca86c"]
Nov 03 21:08:02.354096 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:08:02.354071    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28a7a1c7_5d00_4af8_a0b4_b56fe4d68c2d.slice/crio-06d4a5e307dc7bf305e069a389734415f1049de7e8270983dc5b802d56d40651 WatchSource:0}: Error finding container 06d4a5e307dc7bf305e069a389734415f1049de7e8270983dc5b802d56d40651: Status 404 returned error can't find the container with id 06d4a5e307dc7bf305e069a389734415f1049de7e8270983dc5b802d56d40651
Nov 03 21:08:02.356108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.356088    5318 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:08:02.361645 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.361623    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:02.432476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.430080    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"]
Nov 03 21:08:02.433209 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.433174    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0776c6-frhq8"]
Nov 03 21:08:02.895638 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:02.895601    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:03.104927 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.104882    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:03.265279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.265174    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f6fc77-3c95-4e0a-93fd-99d867f19ce7" path="/var/lib/kubelet/pods/a7f6fc77-3c95-4e0a-93fd-99d867f19ce7/volumes"
Nov 03 21:08:03.268371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.268245    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6927-5ca86c" event={"ID":"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d","Type":"ContainerStarted","Data":"06d4a5e307dc7bf305e069a389734415f1049de7e8270983dc5b802d56d40651"}
Nov 03 21:08:03.285063 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.285035    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:08:03.286174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.286152    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:08:03.287454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.287427    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:08:03.311838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:03.311806    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:05.284540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:05.284488    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6927-5ca86c" event={"ID":"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d","Type":"ContainerStarted","Data":"54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9"}
Nov 03 21:08:05.393381 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:05.393289    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6927-5ca86c" podStartSLOduration=2.404991468 podStartE2EDuration="4.393271146s" totalImagesPullingTime="1.988279678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:05.390818161 +0000 UTC m=+2278.370375001" watchObservedRunningTime="2026-11-03 21:08:05.393271146 +0000 UTC m=+2278.372827987"
Nov 03 21:08:06.620066 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:06.620027    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"]
Nov 03 21:08:06.620588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:06.620225    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" podUID="87415c4c-07b7-4a15-a070-8e1e879d0777" containerName="maas-api" containerID="cri-o://aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9" gracePeriod=30
Nov 03 21:08:06.786790 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:06.786746    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:06.938554 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:06.938512    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6927-5ca86c"]
Nov 03 21:08:06.944501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:06.944475    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6927-5ca86c"]
Nov 03 21:08:06.963378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:06.959751    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 21:08:07.099426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.097286    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\") pod \"87415c4c-07b7-4a15-a070-8e1e879d0777\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") "
Nov 03 21:08:07.099426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.097414    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs\") pod \"87415c4c-07b7-4a15-a070-8e1e879d0777\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") "
Nov 03 21:08:07.099426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.097555    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca\") pod \"87415c4c-07b7-4a15-a070-8e1e879d0777\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") "
Nov 03 21:08:07.099426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.097655    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7\" (UniqueName: \"kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7\") pod \"87415c4c-07b7-4a15-a070-8e1e879d0777\" (UID: \"87415c4c-07b7-4a15-a070-8e1e879d0777\") "
Nov 03 21:08:07.101281 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.101234    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca" pod "87415c4c-07b7-4a15-a070-8e1e879d0777" (UID: "87415c4c-07b7-4a15-a070-8e1e879d0777"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:07.113460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.111783    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls" pod "87415c4c-07b7-4a15-a070-8e1e879d0777" (UID: "87415c4c-07b7-4a15-a070-8e1e879d0777"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:07.113460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.112233    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs" pod "87415c4c-07b7-4a15-a070-8e1e879d0777" (UID: "87415c4c-07b7-4a15-a070-8e1e879d0777"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:07.122960 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.122575    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7" pod "87415c4c-07b7-4a15-a070-8e1e879d0777" (UID: "87415c4c-07b7-4a15-a070-8e1e879d0777"). InnerVolumeSpecName "kube-api-access-zqdc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:07.200090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.199941    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87415c4c-07b7-4a15-a070-8e1e879d0777-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:07.200090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.199978    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqdc7\" (UniqueName: \"kubernetes.io/projected/87415c4c-07b7-4a15-a070-8e1e879d0777-kube-api-access-zqdc7\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:07.200090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.199996    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:07.200090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.200010    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87415c4c-07b7-4a15-a070-8e1e879d0777-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:07.276071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.276041    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:07.301286 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.301257    5318 generic.go:356] "Generic (PLEG): container finished" podID="87415c4c-07b7-4a15-a070-8e1e879d0777" containerID="aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9" exitCode=0
Nov 03 21:08:07.301480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.301303    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"
Nov 03 21:08:07.301480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.301368    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" event={"ID":"87415c4c-07b7-4a15-a070-8e1e879d0777","Type":"ContainerDied","Data":"aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9"}
Nov 03 21:08:07.301480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.301425    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq" event={"ID":"87415c4c-07b7-4a15-a070-8e1e879d0777","Type":"ContainerDied","Data":"d5e2388b7210dfa177a4982628d988367d1547755d5f6f5348ecef1ec1d59721"}
Nov 03 21:08:07.301674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.301487    5318 scope.go:122] "RemoveContainer" containerID="aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9"
Nov 03 21:08:07.301674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.301560    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6927-5ca86c" podUID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" containerName="test-curl-6927-5ca86c" containerID="cri-o://54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9" gracePeriod=2
Nov 03 21:08:07.312027 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.312004    5318 scope.go:122] "RemoveContainer" containerID="aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9"
Nov 03 21:08:07.312321 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:07.312298    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9\": container with ID starting with aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9 not found: ID does not exist" containerID="aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9"
Nov 03 21:08:07.312429 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.312327    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9"} err="failed to get container status \"aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9\": rpc error: code = NotFound desc = could not find container \"aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9\": container with ID starting with aa7484df0b320903f2803435b378e5b67853d9ce19bf9f51d41d6e0e57b230c9 not found: ID does not exist"
Nov 03 21:08:07.460855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.460763    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"]
Nov 03 21:08:07.468615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.468585    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-cc5d0f-5b5c569b88-xw8xq"]
Nov 03 21:08:07.522762 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.522732    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:07.738829 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:07.738724    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:08.680108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:08.680073    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:09.007553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.007470    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:09.211200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.211160    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:09.224553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.224528    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"]
Nov 03 21:08:09.256689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.256552    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87415c4c-07b7-4a15-a070-8e1e879d0777" path="/var/lib/kubelet/pods/87415c4c-07b7-4a15-a070-8e1e879d0777/volumes"
Nov 03 21:08:09.270925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.270869    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-cc5d0f-zz95s"]
Nov 03 21:08:09.548192 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.548165    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:09.703562 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.703494    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:09.740495 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.740470    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287\" (UniqueName: \"kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287\") pod \"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d\" (UID: \"28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d\") "
Nov 03 21:08:09.742606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.742575    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287" pod "28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" (UID: "28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d"). InnerVolumeSpecName "kube-api-access-8k287". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:09.842414 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.842384    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k287\" (UniqueName: \"kubernetes.io/projected/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d-kube-api-access-8k287\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:09.923027 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:09.922986    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:10.265422 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.258581    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:10.323490 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.323457    5318 generic.go:356] "Generic (PLEG): container finished" podID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" containerID="54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9" exitCode=137
Nov 03 21:08:10.323672 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.323540    5318 scope.go:122] "RemoveContainer" containerID="54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9"
Nov 03 21:08:10.323672 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.323638    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:10.336989 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.336969    5318 scope.go:122] "RemoveContainer" containerID="54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9"
Nov 03 21:08:10.337515 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:10.337483    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9\": container with ID starting with 54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9 not found: ID does not exist" containerID="54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9"
Nov 03 21:08:10.337622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.337525    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9"} err="failed to get container status \"54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9\": rpc error: code = NotFound desc = could not find container \"54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9\": container with ID starting with 54eb0ac2aa325bdf3a1bd4c858c70555a6f0a5986ebe1789f2167abdb9860cb9 not found: ID does not exist"
Nov 03 21:08:10.385362 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:10.385303    5318 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6927-5ca86c\" is forbidden: User \"system:node:ip-10-0-135-141.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-141.us-west-2.compute.internal' and this object" podUID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" pod="opendatahub/test-curl-6927-5ca86c"
Nov 03 21:08:10.767538 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:10.767502    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:11.256726 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.256694    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" path="/var/lib/kubelet/pods/28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d/volumes"
Nov 03 21:08:11.257155 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.257141    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7fab6a-f141-4651-9376-3b80b3fe93c5" path="/var/lib/kubelet/pods/9f7fab6a-f141-4651-9376-3b80b3fe93c5/volumes"
Nov 03 21:08:11.359822 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.359777    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:11.533870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.533784    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"]
Nov 03 21:08:11.534372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.534355    5318 cpu_manager.go:414] "containerMap: removing container" podUID="87415c4c-07b7-4a15-a070-8e1e879d0777" containerName="maas-api"
Nov 03 21:08:11.534418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.534375    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87415c4c-07b7-4a15-a070-8e1e879d0777" containerName="maas-api"
Nov 03 21:08:11.534418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.534416    5318 cpu_manager.go:414] "containerMap: removing container" podUID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" containerName="test-curl-6927-5ca86c"
Nov 03 21:08:11.534540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.534422    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" containerName="test-curl-6927-5ca86c"
Nov 03 21:08:11.534540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.534534    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="87415c4c-07b7-4a15-a070-8e1e879d0777" containerName="maas-api"
Nov 03 21:08:11.534606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.534545    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a7a1c7-5d00-4af8-a0b4-b56fe4d68c2d" containerName="test-curl-6927-5ca86c"
Nov 03 21:08:11.539782 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.539761    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.565523 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.565493    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.565674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.565599    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.565754 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.565707    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.612498 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.612460    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:11.613354 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.613317    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"]
Nov 03 21:08:11.651886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.651860    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:11.652088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.652055    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:08:11.667027 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.666992    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.667202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.667149    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.667262 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.667247    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.667992 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.667967    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.669805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.669782    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.709515 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.709485    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"]
Nov 03 21:08:11.713708 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.713687    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.730441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.730413    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g\") pod \"maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.763936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.763907    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fc4e8d\""
Nov 03 21:08:11.764168 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.764151    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fc4e8d-dockercfg-lwrds\""
Nov 03 21:08:11.768869 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.768848    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmdq\" (UniqueName: \"kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq\") pod \"payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.768974 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.768956    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.781528 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.781504    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"]
Nov 03 21:08:11.853808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.852561    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:11.867418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.863676    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"]
Nov 03 21:08:11.879045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.873876    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmdq\" (UniqueName: \"kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq\") pod \"payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.879045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.874024    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.879045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.875068    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.893622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.893592    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:11.938689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.938568    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"]
Nov 03 21:08:11.947973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.947908    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmdq\" (UniqueName: \"kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq\") pod \"payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:11.975441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.975403    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwvs\" (UniqueName: \"kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs\") pod \"payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:11.975672 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:11.975655    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:12.027816 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.027785    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:12.079025 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.077277    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwvs\" (UniqueName: \"kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs\") pod \"payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:12.079025 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.077597    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:12.079241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.079111    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:12.136153 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.136075    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwvs\" (UniqueName: \"kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs\") pod \"payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:12.136542 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.136515    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:12.279626 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.279589    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:12.310503 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.310439    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"]
Nov 03 21:08:12.312980 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:08:12.312952    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f522098_dd68_4c8e_86d8_fc34725d54af.slice/crio-4dd4c50bfb6166ec6cb4e85465c85f9db8e5c0e9b98a257aead45ee432c35673 WatchSource:0}: Error finding container 4dd4c50bfb6166ec6cb4e85465c85f9db8e5c0e9b98a257aead45ee432c35673: Status 404 returned error can't find the container with id 4dd4c50bfb6166ec6cb4e85465c85f9db8e5c0e9b98a257aead45ee432c35673
Nov 03 21:08:12.343543 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.343508    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx" event={"ID":"5f522098-dd68-4c8e-86d8-fc34725d54af","Type":"ContainerStarted","Data":"4dd4c50bfb6166ec6cb4e85465c85f9db8e5c0e9b98a257aead45ee432c35673"}
Nov 03 21:08:12.417147 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.417112    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"]
Nov 03 21:08:12.417866 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:08:12.417838    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77acde91_fa20_44cc_9b8d_bfa0ff436811.slice/crio-dd2216d7065b8964513817a68a57079b916f9ac69b379fa135e328e40a5fbac9 WatchSource:0}: Error finding container dd2216d7065b8964513817a68a57079b916f9ac69b379fa135e328e40a5fbac9: Status 404 returned error can't find the container with id dd2216d7065b8964513817a68a57079b916f9ac69b379fa135e328e40a5fbac9
Nov 03 21:08:12.652301 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:12.652274    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"]
Nov 03 21:08:12.653963 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:08:12.653931    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739c317f_11d8_4a9b_bfa9_053dcca3e36c.slice/crio-41b1aa9cd4ce7eda0be822cc67dab691888878e45e3dd8921dcd5fedf3b35a17 WatchSource:0}: Error finding container 41b1aa9cd4ce7eda0be822cc67dab691888878e45e3dd8921dcd5fedf3b35a17: Status 404 returned error can't find the container with id 41b1aa9cd4ce7eda0be822cc67dab691888878e45e3dd8921dcd5fedf3b35a17
Nov 03 21:08:13.352052 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.352012    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9" event={"ID":"739c317f-11d8-4a9b-bfa9-053dcca3e36c","Type":"ContainerStarted","Data":"8d7f24521a7c503206f86fe86455d9fa997317ddb3b37a344a0b6eb9116ba288"}
Nov 03 21:08:13.352220 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.352078    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9" event={"ID":"739c317f-11d8-4a9b-bfa9-053dcca3e36c","Type":"ContainerStarted","Data":"41b1aa9cd4ce7eda0be822cc67dab691888878e45e3dd8921dcd5fedf3b35a17"}
Nov 03 21:08:13.352220 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.352107    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:13.353843 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.353816    5318 generic.go:356] "Generic (PLEG): container finished" podID="5f522098-dd68-4c8e-86d8-fc34725d54af" containerID="04c86674c09aaee0f9756cf739e2a87d029e0b15858381f3396eec771d896fcb" exitCode=0
Nov 03 21:08:13.353952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.353889    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx" event={"ID":"5f522098-dd68-4c8e-86d8-fc34725d54af","Type":"ContainerDied","Data":"04c86674c09aaee0f9756cf739e2a87d029e0b15858381f3396eec771d896fcb"}
Nov 03 21:08:13.355851 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.355827    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" event={"ID":"77acde91-fa20-44cc-9b8d-bfa0ff436811","Type":"ContainerStarted","Data":"06b7ad407ca8ec58bc747c9b90e3e459638e3f793907cb4389888c62579eea9d"}
Nov 03 21:08:13.355951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.355884    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" event={"ID":"77acde91-fa20-44cc-9b8d-bfa0ff436811","Type":"ContainerStarted","Data":"dd2216d7065b8964513817a68a57079b916f9ac69b379fa135e328e40a5fbac9"}
Nov 03 21:08:13.355951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.355915    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:13.499254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.499154    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9" podStartSLOduration=2.499141076 podStartE2EDuration="2.499141076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:13.49846559 +0000 UTC m=+2286.478022431" watchObservedRunningTime="2026-11-03 21:08:13.499141076 +0000 UTC m=+2286.478697916"
Nov 03 21:08:13.715775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:13.715724    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" podStartSLOduration=2.715710424 podStartE2EDuration="2.715710424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:13.714080868 +0000 UTC m=+2286.693637708" watchObservedRunningTime="2026-11-03 21:08:13.715710424 +0000 UTC m=+2286.695267263"
Nov 03 21:08:14.525238 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.525212    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:14.610115 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.610082    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca\") pod \"5f522098-dd68-4c8e-86d8-fc34725d54af\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") "
Nov 03 21:08:14.610279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.610208    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca\") pod \"5f522098-dd68-4c8e-86d8-fc34725d54af\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") "
Nov 03 21:08:14.610279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.610260    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g\") pod \"5f522098-dd68-4c8e-86d8-fc34725d54af\" (UID: \"5f522098-dd68-4c8e-86d8-fc34725d54af\") "
Nov 03 21:08:14.610517 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.610491    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca" pod "5f522098-dd68-4c8e-86d8-fc34725d54af" (UID: "5f522098-dd68-4c8e-86d8-fc34725d54af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:14.610823 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.610802    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:14.612488 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.612441    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g" pod "5f522098-dd68-4c8e-86d8-fc34725d54af" (UID: "5f522098-dd68-4c8e-86d8-fc34725d54af"). InnerVolumeSpecName "kube-api-access-59b2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:14.612488 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.612445    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca" pod "5f522098-dd68-4c8e-86d8-fc34725d54af" (UID: "5f522098-dd68-4c8e-86d8-fc34725d54af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:14.707195 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.707157    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:14.711763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.711737    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f522098-dd68-4c8e-86d8-fc34725d54af-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:14.711763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:14.711761    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59b2g\" (UniqueName: \"kubernetes.io/projected/5f522098-dd68-4c8e-86d8-fc34725d54af-kube-api-access-59b2g\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:15.369136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:15.369110    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"
Nov 03 21:08:15.369319 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:15.369149    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx" event={"ID":"5f522098-dd68-4c8e-86d8-fc34725d54af","Type":"ContainerDied","Data":"4dd4c50bfb6166ec6cb4e85465c85f9db8e5c0e9b98a257aead45ee432c35673"}
Nov 03 21:08:15.369319 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:15.369197    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dd4c50bfb6166ec6cb4e85465c85f9db8e5c0e9b98a257aead45ee432c35673"
Nov 03 21:08:15.670442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:15.670411    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"]
Nov 03 21:08:15.708644 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:15.708610    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"]
Nov 03 21:08:16.168503 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:16.168474    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:16.382462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:16.382428    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:18.633814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:18.633778    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:19.804658 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:19.804627    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"]
Nov 03 21:08:19.914629 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:19.914594    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"]
Nov 03 21:08:20.159610 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:20.159561    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:20.365178 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:20.365110    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:20.580836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:20.580712    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:22.776518 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:22.776482    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:22.982137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:22.982105    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:24.365578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:24.365540    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:08:24.365964 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:24.365627    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:08:24.805332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:24.805241    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw" podUID="545898ed-960d-4a1f-9dd9-44fe5d666f78" containerName="payload-processing" containerID="cri-o://942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d" gracePeriod=25
Nov 03 21:08:24.915774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:24.915711    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586" podUID="ed70b30c-5558-427b-bd32-d78ffb7a1a8e" containerName="payload-pre-processing" containerID="cri-o://69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8" gracePeriod=25
Nov 03 21:08:25.095779 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.095749    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:08:25.210166 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.210135    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:08:25.271900 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.271859    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume\") pod \"545898ed-960d-4a1f-9dd9-44fe5d666f78\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") "
Nov 03 21:08:25.272087 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.271916    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt\" (UniqueName: \"kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt\") pod \"545898ed-960d-4a1f-9dd9-44fe5d666f78\" (UID: \"545898ed-960d-4a1f-9dd9-44fe5d666f78\") "
Nov 03 21:08:25.272329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.272301    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume" pod "545898ed-960d-4a1f-9dd9-44fe5d666f78" (UID: "545898ed-960d-4a1f-9dd9-44fe5d666f78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:25.274068 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.274042    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt" pod "545898ed-960d-4a1f-9dd9-44fe5d666f78" (UID: "545898ed-960d-4a1f-9dd9-44fe5d666f78"). InnerVolumeSpecName "kube-api-access-42ktt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:25.373102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.373022    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22\" (UniqueName: \"kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22\") pod \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") "
Nov 03 21:08:25.373506 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.373158    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume\") pod \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\" (UID: \"ed70b30c-5558-427b-bd32-d78ffb7a1a8e\") "
Nov 03 21:08:25.373611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.373588    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545898ed-960d-4a1f-9dd9-44fe5d666f78-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.373652 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.373624    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ktt\" (UniqueName: \"kubernetes.io/projected/545898ed-960d-4a1f-9dd9-44fe5d666f78-kube-api-access-42ktt\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.373652 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.373589    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume" pod "ed70b30c-5558-427b-bd32-d78ffb7a1a8e" (UID: "ed70b30c-5558-427b-bd32-d78ffb7a1a8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:25.375222 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.375199    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22" pod "ed70b30c-5558-427b-bd32-d78ffb7a1a8e" (UID: "ed70b30c-5558-427b-bd32-d78ffb7a1a8e"). InnerVolumeSpecName "kube-api-access-lpk22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:25.436004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.435967    5318 generic.go:356] "Generic (PLEG): container finished" podID="545898ed-960d-4a1f-9dd9-44fe5d666f78" containerID="942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d" exitCode=0
Nov 03 21:08:25.436197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.436015    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"
Nov 03 21:08:25.436197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.436055    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw" event={"ID":"545898ed-960d-4a1f-9dd9-44fe5d666f78","Type":"ContainerDied","Data":"942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d"}
Nov 03 21:08:25.436197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.436108    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw" event={"ID":"545898ed-960d-4a1f-9dd9-44fe5d666f78","Type":"ContainerDied","Data":"537a4f62a6720a9f4e0cdd7b5025e3ea2d7f9f031dce63fe1abfc6bc79139412"}
Nov 03 21:08:25.436197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.436129    5318 scope.go:122] "RemoveContainer" containerID="942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d"
Nov 03 21:08:25.437604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.437584    5318 generic.go:356] "Generic (PLEG): container finished" podID="ed70b30c-5558-427b-bd32-d78ffb7a1a8e" containerID="69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8" exitCode=0
Nov 03 21:08:25.437715 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.437632    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"
Nov 03 21:08:25.437715 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.437655    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586" event={"ID":"ed70b30c-5558-427b-bd32-d78ffb7a1a8e","Type":"ContainerDied","Data":"69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8"}
Nov 03 21:08:25.437715 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.437694    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586" event={"ID":"ed70b30c-5558-427b-bd32-d78ffb7a1a8e","Type":"ContainerDied","Data":"d3a7c685eaf95e2cab513ad1d2c9392b6d443e3d46516771f8c92706dc487c8b"}
Nov 03 21:08:25.447877 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.447856    5318 scope.go:122] "RemoveContainer" containerID="942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d"
Nov 03 21:08:25.448159 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:25.448138    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d\": container with ID starting with 942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d not found: ID does not exist" containerID="942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d"
Nov 03 21:08:25.448229 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.448166    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d"} err="failed to get container status \"942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d\": rpc error: code = NotFound desc = could not find container \"942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d\": container with ID starting with 942cde34216a25c62e559ffd35066f5c33b5e76cfdafe84f570efd74c487585d not found: ID does not exist"
Nov 03 21:08:25.448229 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.448183    5318 scope.go:122] "RemoveContainer" containerID="69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8"
Nov 03 21:08:25.457232 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.457215    5318 scope.go:122] "RemoveContainer" containerID="69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8"
Nov 03 21:08:25.457512 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:25.457489    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8\": container with ID starting with 69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8 not found: ID does not exist" containerID="69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8"
Nov 03 21:08:25.457580 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.457519    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8"} err="failed to get container status \"69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8\": rpc error: code = NotFound desc = could not find container \"69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8\": container with ID starting with 69c05e0849ec683a222747f7b5920f977791c676c8a26eb9e7ec42a4b78bbba8 not found: ID does not exist"
Nov 03 21:08:25.475325 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.475296    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpk22\" (UniqueName: \"kubernetes.io/projected/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-kube-api-access-lpk22\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.475325 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.475324    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70b30c-5558-427b-bd32-d78ffb7a1a8e-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:25.591795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.591699    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"]
Nov 03 21:08:25.597913 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.597883    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-tc586"]
Nov 03 21:08:25.679859 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.679744    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:25.699954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.699694    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"]
Nov 03 21:08:25.702400 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:25.702377    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-qnczw"]
Nov 03 21:08:25.934134 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:25.934052    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:08:26.200813 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:26.200696    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:27.255578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:27.255546    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545898ed-960d-4a1f-9dd9-44fe5d666f78" path="/var/lib/kubelet/pods/545898ed-960d-4a1f-9dd9-44fe5d666f78/volumes"
Nov 03 21:08:27.255991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:27.255976    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed70b30c-5558-427b-bd32-d78ffb7a1a8e" path="/var/lib/kubelet/pods/ed70b30c-5558-427b-bd32-d78ffb7a1a8e/volumes"
Nov 03 21:08:28.975636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:28.975596    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:30.548966 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:30.548929    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:30.691769 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:30.691722    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" containerID="cri-o://1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478" gracePeriod=45
Nov 03 21:08:30.729224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:30.729173    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main" containerID="cri-o://31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4" gracePeriod=45
Nov 03 21:08:30.753799 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:30.753758    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused"
Nov 03 21:08:30.757531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:30.757508    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:31.026553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.026528    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 21:08:31.029860 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.029837    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 21:08:31.046113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046090    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir\") pod \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") "
Nov 03 21:08:31.046230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046154    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs\") pod \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") "
Nov 03 21:08:31.046230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046179    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache\") pod \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") "
Nov 03 21:08:31.046230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046213    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q\") pod \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") "
Nov 03 21:08:31.046419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046277    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir\") pod \"0c9222a3-7d8e-474d-8f67-04804f48aaac\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") "
Nov 03 21:08:31.046419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046321    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache\") pod \"0c9222a3-7d8e-474d-8f67-04804f48aaac\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") "
Nov 03 21:08:31.046419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046384    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm\") pod \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") "
Nov 03 21:08:31.046572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046416    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir" pod "c0577730-0066-4a22-9ab8-bb2b6d1fde11" (UID: "c0577730-0066-4a22-9ab8-bb2b6d1fde11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.046572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046441    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs\") pod \"0c9222a3-7d8e-474d-8f67-04804f48aaac\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") "
Nov 03 21:08:31.046572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046511    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29\" (UniqueName: \"kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29\") pod \"0c9222a3-7d8e-474d-8f67-04804f48aaac\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") "
Nov 03 21:08:31.046730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046570    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm\") pod \"0c9222a3-7d8e-474d-8f67-04804f48aaac\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") "
Nov 03 21:08:31.046730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046529    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir" pod "0c9222a3-7d8e-474d-8f67-04804f48aaac" (UID: "0c9222a3-7d8e-474d-8f67-04804f48aaac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.046730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046599    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home\") pod \"0c9222a3-7d8e-474d-8f67-04804f48aaac\" (UID: \"0c9222a3-7d8e-474d-8f67-04804f48aaac\") "
Nov 03 21:08:31.046730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046633    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home\") pod \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\" (UID: \"c0577730-0066-4a22-9ab8-bb2b6d1fde11\") "
Nov 03 21:08:31.046730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046635    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache" pod "0c9222a3-7d8e-474d-8f67-04804f48aaac" (UID: "0c9222a3-7d8e-474d-8f67-04804f48aaac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.047042 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046819    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache" pod "c0577730-0066-4a22-9ab8-bb2b6d1fde11" (UID: "c0577730-0066-4a22-9ab8-bb2b6d1fde11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.047042 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046827    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home" pod "c0577730-0066-4a22-9ab8-bb2b6d1fde11" (UID: "c0577730-0066-4a22-9ab8-bb2b6d1fde11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.047042 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.046978    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home" pod "0c9222a3-7d8e-474d-8f67-04804f48aaac" (UID: "0c9222a3-7d8e-474d-8f67-04804f48aaac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.047236 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.047217    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.047295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.047241    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.047295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.047255    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.047295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.047267    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.047295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.047282    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.047528 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.047295    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.049222 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.049198    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs" pod "c0577730-0066-4a22-9ab8-bb2b6d1fde11" (UID: "c0577730-0066-4a22-9ab8-bb2b6d1fde11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:31.049361 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.049314    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29" pod "0c9222a3-7d8e-474d-8f67-04804f48aaac" (UID: "0c9222a3-7d8e-474d-8f67-04804f48aaac"). InnerVolumeSpecName "kube-api-access-b4m29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:31.049361 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.049323    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs" pod "0c9222a3-7d8e-474d-8f67-04804f48aaac" (UID: "0c9222a3-7d8e-474d-8f67-04804f48aaac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:31.049472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.049377    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q" pod "c0577730-0066-4a22-9ab8-bb2b6d1fde11" (UID: "c0577730-0066-4a22-9ab8-bb2b6d1fde11"). InnerVolumeSpecName "kube-api-access-c4t9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:31.049472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.049406    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm" pod "0c9222a3-7d8e-474d-8f67-04804f48aaac" (UID: "0c9222a3-7d8e-474d-8f67-04804f48aaac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.049535 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.049493    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm" pod "c0577730-0066-4a22-9ab8-bb2b6d1fde11" (UID: "c0577730-0066-4a22-9ab8-bb2b6d1fde11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:31.149016 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.148978    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0577730-0066-4a22-9ab8-bb2b6d1fde11-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.149016 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.149018    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9222a3-7d8e-474d-8f67-04804f48aaac-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.149255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.149034    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4m29\" (UniqueName: \"kubernetes.io/projected/0c9222a3-7d8e-474d-8f67-04804f48aaac-kube-api-access-b4m29\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.149255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.149050    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c9222a3-7d8e-474d-8f67-04804f48aaac-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.149255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.149063    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0577730-0066-4a22-9ab8-bb2b6d1fde11-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.149255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.149076    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/c0577730-0066-4a22-9ab8-bb2b6d1fde11-kube-api-access-c4t9q\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:31.477565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.477478    5318 generic.go:356] "Generic (PLEG): container finished" podID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerID="31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4" exitCode=0
Nov 03 21:08:31.477565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.477537    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"
Nov 03 21:08:31.477770 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.477537    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" event={"ID":"c0577730-0066-4a22-9ab8-bb2b6d1fde11","Type":"ContainerDied","Data":"31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4"}
Nov 03 21:08:31.477770 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.477699    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg" event={"ID":"c0577730-0066-4a22-9ab8-bb2b6d1fde11","Type":"ContainerDied","Data":"faa5dabe1175f39244948fb17433b9a34148a3ba966bc8ece38f883d434f94a3"}
Nov 03 21:08:31.477770 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.477730    5318 scope.go:122] "RemoveContainer" containerID="31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4"
Nov 03 21:08:31.479057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.479034    5318 generic.go:356] "Generic (PLEG): container finished" podID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerID="1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478" exitCode=0
Nov 03 21:08:31.479141 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.479125    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"
Nov 03 21:08:31.479203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.479166    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" event={"ID":"0c9222a3-7d8e-474d-8f67-04804f48aaac","Type":"ContainerDied","Data":"1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478"}
Nov 03 21:08:31.479203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.479193    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp" event={"ID":"0c9222a3-7d8e-474d-8f67-04804f48aaac","Type":"ContainerDied","Data":"97115e0b0d4a82d35e6dddae984754a554b00a0f6bb2ea43bdb868d76bcf6612"}
Nov 03 21:08:31.488709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.488689    5318 scope.go:122] "RemoveContainer" containerID="31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4"
Nov 03 21:08:31.489013 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:31.488990    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4\": container with ID starting with 31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4 not found: ID does not exist" containerID="31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4"
Nov 03 21:08:31.489113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.489027    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4"} err="failed to get container status \"31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4\": rpc error: code = NotFound desc = could not find container \"31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4\": container with ID starting with 31b1620383d9cb40e2d5cde0aaef94e61fe57a57b48fde48285702051071c4b4 not found: ID does not exist"
Nov 03 21:08:31.489113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.489049    5318 scope.go:122] "RemoveContainer" containerID="1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478"
Nov 03 21:08:31.497794 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.497768    5318 scope.go:122] "RemoveContainer" containerID="1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478"
Nov 03 21:08:31.498118 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:31.498096    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478\": container with ID starting with 1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478 not found: ID does not exist" containerID="1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478"
Nov 03 21:08:31.498197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.498126    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478"} err="failed to get container status \"1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478\": rpc error: code = NotFound desc = could not find container \"1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478\": container with ID starting with 1ee18de8247e7ec3bb4a4543276ea702e3df5714d687ab2193a02502a5310478 not found: ID does not exist"
Nov 03 21:08:31.636140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.636105    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"]
Nov 03 21:08:31.640411 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.640385    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/distinct-2-w4-3d2008-kserve-6cb6f5b747-5rqdp"]
Nov 03 21:08:31.742520 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.742442    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"]
Nov 03 21:08:31.747937 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:31.747914    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3d2008/premium-w4-3d2008-kserve-5469fcb47c-kkgcg"]
Nov 03 21:08:33.263743 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:33.263708    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" path="/var/lib/kubelet/pods/0c9222a3-7d8e-474d-8f67-04804f48aaac/volumes"
Nov 03 21:08:33.264246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:33.264229    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" path="/var/lib/kubelet/pods/c0577730-0066-4a22-9ab8-bb2b6d1fde11/volumes"
Nov 03 21:08:33.358768 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:33.358729    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"]
Nov 03 21:08:33.469432 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:33.469396    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"]
Nov 03 21:08:33.800088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:33.800051    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:34.006382 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:34.006331    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:35.997360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:35.996885    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:38.359810 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.359773    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq" podUID="93681b1e-6714-4daa-855e-ca85e7eae2d0" containerName="payload-processing" containerID="cri-o://9e55ac8bb65cad945bc642c1c63951897c81ea9b2108261c31832b8fa5368edc" gracePeriod=25
Nov 03 21:08:38.469950 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.469886    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl" podUID="500d5ae6-802c-4a41-a9be-68ce83661b8d" containerName="payload-pre-processing" containerID="cri-o://1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed" gracePeriod=25
Nov 03 21:08:38.581620 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.581574    5318 generic.go:356] "Generic (PLEG): container finished" podID="93681b1e-6714-4daa-855e-ca85e7eae2d0" containerID="9e55ac8bb65cad945bc642c1c63951897c81ea9b2108261c31832b8fa5368edc" exitCode=0
Nov 03 21:08:38.581947 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.581830    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq" event={"ID":"93681b1e-6714-4daa-855e-ca85e7eae2d0","Type":"ContainerDied","Data":"9e55ac8bb65cad945bc642c1c63951897c81ea9b2108261c31832b8fa5368edc"}
Nov 03 21:08:38.647841 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.647814    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 21:08:38.674424 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.674390    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume\") pod \"93681b1e-6714-4daa-855e-ca85e7eae2d0\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") "
Nov 03 21:08:38.674572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.674557    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms\" (UniqueName: \"kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms\") pod \"93681b1e-6714-4daa-855e-ca85e7eae2d0\" (UID: \"93681b1e-6714-4daa-855e-ca85e7eae2d0\") "
Nov 03 21:08:38.674782 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.674757    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume" pod "93681b1e-6714-4daa-855e-ca85e7eae2d0" (UID: "93681b1e-6714-4daa-855e-ca85e7eae2d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:38.675243 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.675219    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93681b1e-6714-4daa-855e-ca85e7eae2d0-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:38.678403 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.678363    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms" pod "93681b1e-6714-4daa-855e-ca85e7eae2d0" (UID: "93681b1e-6714-4daa-855e-ca85e7eae2d0"). InnerVolumeSpecName "kube-api-access-pwhms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:38.741432 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.741406    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 21:08:38.775942 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.775914    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume\") pod \"500d5ae6-802c-4a41-a9be-68ce83661b8d\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") "
Nov 03 21:08:38.776076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.776008    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt\" (UniqueName: \"kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt\") pod \"500d5ae6-802c-4a41-a9be-68ce83661b8d\" (UID: \"500d5ae6-802c-4a41-a9be-68ce83661b8d\") "
Nov 03 21:08:38.776382 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.776320    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume" pod "500d5ae6-802c-4a41-a9be-68ce83661b8d" (UID: "500d5ae6-802c-4a41-a9be-68ce83661b8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:38.776845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.776697    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/500d5ae6-802c-4a41-a9be-68ce83661b8d-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:38.776845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.776728    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhms\" (UniqueName: \"kubernetes.io/projected/93681b1e-6714-4daa-855e-ca85e7eae2d0-kube-api-access-pwhms\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:38.778149 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.778125    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt" pod "500d5ae6-802c-4a41-a9be-68ce83661b8d" (UID: "500d5ae6-802c-4a41-a9be-68ce83661b8d"). InnerVolumeSpecName "kube-api-access-vb5dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:38.878019 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:38.877952    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5dt\" (UniqueName: \"kubernetes.io/projected/500d5ae6-802c-4a41-a9be-68ce83661b8d-kube-api-access-vb5dt\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:39.590577 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.590546    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"
Nov 03 21:08:39.591029 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.590575    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq" event={"ID":"93681b1e-6714-4daa-855e-ca85e7eae2d0","Type":"ContainerDied","Data":"23f9403c890bb9c60c4a90b1174b27c7c5eaf011768b9e607b0c018fe877f093"}
Nov 03 21:08:39.591029 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.590649    5318 scope.go:122] "RemoveContainer" containerID="9e55ac8bb65cad945bc642c1c63951897c81ea9b2108261c31832b8fa5368edc"
Nov 03 21:08:39.592557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.592530    5318 generic.go:356] "Generic (PLEG): container finished" podID="500d5ae6-802c-4a41-a9be-68ce83661b8d" containerID="1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed" exitCode=0
Nov 03 21:08:39.592696 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.592570    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"
Nov 03 21:08:39.592696 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.592607    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl" event={"ID":"500d5ae6-802c-4a41-a9be-68ce83661b8d","Type":"ContainerDied","Data":"1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed"}
Nov 03 21:08:39.592696 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.592658    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl" event={"ID":"500d5ae6-802c-4a41-a9be-68ce83661b8d","Type":"ContainerDied","Data":"5ba19ca356943fc5e71f1e08b58ec797f5d7ff9a1532785588725aa14793bb0b"}
Nov 03 21:08:39.602560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.602540    5318 scope.go:122] "RemoveContainer" containerID="1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed"
Nov 03 21:08:39.613012 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.612981    5318 scope.go:122] "RemoveContainer" containerID="1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed"
Nov 03 21:08:39.613284 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:39.613265    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed\": container with ID starting with 1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed not found: ID does not exist" containerID="1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed"
Nov 03 21:08:39.613391 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.613291    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed"} err="failed to get container status \"1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed\": rpc error: code = NotFound desc = could not find container \"1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed\": container with ID starting with 1e8e9f7a1e7f64d2d6bd4e4773284dd42b8e9a70020ecbdff82246774d45f4ed not found: ID does not exist"
Nov 03 21:08:39.743990 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.743963    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"]
Nov 03 21:08:39.790314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.790285    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9ce672-54bd578699-ss9nl"]
Nov 03 21:08:39.849572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.849511    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"]
Nov 03 21:08:39.853990 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:39.853966    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-9ce672-b8d9d7d45-gmdwq"]
Nov 03 21:08:40.310712 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.310637    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"]
Nov 03 21:08:40.311439 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311419    5318 cpu_manager.go:414] "containerMap: removing container" podUID="500d5ae6-802c-4a41-a9be-68ce83661b8d" containerName="payload-pre-processing"
Nov 03 21:08:40.311530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311439    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500d5ae6-802c-4a41-a9be-68ce83661b8d" containerName="payload-pre-processing"
Nov 03 21:08:40.311530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311456    5318 cpu_manager.go:414] "containerMap: removing container" podUID="93681b1e-6714-4daa-855e-ca85e7eae2d0" containerName="payload-processing"
Nov 03 21:08:40.311530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311463    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93681b1e-6714-4daa-855e-ca85e7eae2d0" containerName="payload-processing"
Nov 03 21:08:40.311530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311494    5318 cpu_manager.go:414] "containerMap: removing container" podUID="ed70b30c-5558-427b-bd32-d78ffb7a1a8e" containerName="payload-pre-processing"
Nov 03 21:08:40.311530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311502    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed70b30c-5558-427b-bd32-d78ffb7a1a8e" containerName="payload-pre-processing"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311537    5318 cpu_manager.go:414] "containerMap: removing container" podUID="545898ed-960d-4a1f-9dd9-44fe5d666f78" containerName="payload-processing"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311545    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545898ed-960d-4a1f-9dd9-44fe5d666f78" containerName="payload-processing"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311561    5318 cpu_manager.go:414] "containerMap: removing container" podUID="5f522098-dd68-4c8e-86d8-fc34725d54af" containerName="revoke-keys"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311567    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f522098-dd68-4c8e-86d8-fc34725d54af" containerName="revoke-keys"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311578    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311584    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311600    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main"
Nov 03 21:08:40.311706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311606    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311744    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9222a3-7d8e-474d-8f67-04804f48aaac" containerName="main"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311760    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0577730-0066-4a22-9ab8-bb2b6d1fde11" containerName="main"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311771    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="500d5ae6-802c-4a41-a9be-68ce83661b8d" containerName="payload-pre-processing"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311779    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="93681b1e-6714-4daa-855e-ca85e7eae2d0" containerName="payload-processing"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311791    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed70b30c-5558-427b-bd32-d78ffb7a1a8e" containerName="payload-pre-processing"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311804    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="545898ed-960d-4a1f-9dd9-44fe5d666f78" containerName="payload-processing"
Nov 03 21:08:40.311939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.311812    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f522098-dd68-4c8e-86d8-fc34725d54af" containerName="revoke-keys"
Nov 03 21:08:40.316881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.316864    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.369849 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.369825    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:08:40.369970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.369945    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:08:40.371835 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.371813    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"]
Nov 03 21:08:40.395680 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.395656    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.395794 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.395699    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78th\" (UniqueName: \"kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.395936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.395911    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.497038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.497001    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.497234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.497060    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78th\" (UniqueName: \"kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.497234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.497180    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.497822 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.497803    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.499394 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.499372    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.554756 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.554730    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78th\" (UniqueName: \"kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th\") pod \"maas-api-revoke-sub-c8532db3-7rbxh\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:40.631001 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:40.630905    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:41.000388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.000325    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"]
Nov 03 21:08:41.003531 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:08:41.003500    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5773541_4239_4aa9_b981_3939bcc5dbcc.slice/crio-088b5c13f4173163a83e54a2400ad76c8fc610d69a6bac92d013dceca14f4ec0 WatchSource:0}: Error finding container 088b5c13f4173163a83e54a2400ad76c8fc610d69a6bac92d013dceca14f4ec0: Status 404 returned error can't find the container with id 088b5c13f4173163a83e54a2400ad76c8fc610d69a6bac92d013dceca14f4ec0
Nov 03 21:08:41.254936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.254867    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500d5ae6-802c-4a41-a9be-68ce83661b8d" path="/var/lib/kubelet/pods/500d5ae6-802c-4a41-a9be-68ce83661b8d/volumes"
Nov 03 21:08:41.255264 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.255251    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93681b1e-6714-4daa-855e-ca85e7eae2d0" path="/var/lib/kubelet/pods/93681b1e-6714-4daa-855e-ca85e7eae2d0/volumes"
Nov 03 21:08:41.298416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.298386    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:41.612382 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.612317    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh" event={"ID":"a5773541-4239-4aa9-b981-3939bcc5dbcc","Type":"ContainerStarted","Data":"d1e52d685c629edbee9a48603cd00f2523cba5766f34b891481af88b8622bb72"}
Nov 03 21:08:41.612565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.612397    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh" event={"ID":"a5773541-4239-4aa9-b981-3939bcc5dbcc","Type":"ContainerStarted","Data":"088b5c13f4173163a83e54a2400ad76c8fc610d69a6bac92d013dceca14f4ec0"}
Nov 03 21:08:41.720951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:41.720903    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh" podStartSLOduration=1.7208902689999999 podStartE2EDuration="1.720890269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:41.720193149 +0000 UTC m=+2314.699749988" watchObservedRunningTime="2026-11-03 21:08:41.720890269 +0000 UTC m=+2314.700447108"
Nov 03 21:08:42.621191 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:42.621154    5318 generic.go:356] "Generic (PLEG): container finished" podID="a5773541-4239-4aa9-b981-3939bcc5dbcc" containerID="d1e52d685c629edbee9a48603cd00f2523cba5766f34b891481af88b8622bb72" exitCode=0
Nov 03 21:08:42.621386 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:42.621227    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh" event={"ID":"a5773541-4239-4aa9-b981-3939bcc5dbcc","Type":"ContainerDied","Data":"d1e52d685c629edbee9a48603cd00f2523cba5766f34b891481af88b8622bb72"}
Nov 03 21:08:43.786559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.786528    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:43.844497 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.844467    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca\") pod \"a5773541-4239-4aa9-b981-3939bcc5dbcc\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") "
Nov 03 21:08:43.844611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.844524    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca\") pod \"a5773541-4239-4aa9-b981-3939bcc5dbcc\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") "
Nov 03 21:08:43.844660 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.844639    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th\" (UniqueName: \"kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th\") pod \"a5773541-4239-4aa9-b981-3939bcc5dbcc\" (UID: \"a5773541-4239-4aa9-b981-3939bcc5dbcc\") "
Nov 03 21:08:43.844838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.844813    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca" pod "a5773541-4239-4aa9-b981-3939bcc5dbcc" (UID: "a5773541-4239-4aa9-b981-3939bcc5dbcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:43.845272 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.845252    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:43.846465 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.846440    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca" pod "a5773541-4239-4aa9-b981-3939bcc5dbcc" (UID: "a5773541-4239-4aa9-b981-3939bcc5dbcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:08:43.846665 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.846641    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th" pod "a5773541-4239-4aa9-b981-3939bcc5dbcc" (UID: "a5773541-4239-4aa9-b981-3939bcc5dbcc"). InnerVolumeSpecName "kube-api-access-k78th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:43.946717 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.946635    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5773541-4239-4aa9-b981-3939bcc5dbcc-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:43.946717 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:43.946667    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78th\" (UniqueName: \"kubernetes.io/projected/a5773541-4239-4aa9-b981-3939bcc5dbcc-kube-api-access-k78th\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:44.636598 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:44.636563    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"
Nov 03 21:08:44.636776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:44.636595    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh" event={"ID":"a5773541-4239-4aa9-b981-3939bcc5dbcc","Type":"ContainerDied","Data":"088b5c13f4173163a83e54a2400ad76c8fc610d69a6bac92d013dceca14f4ec0"}
Nov 03 21:08:44.636776 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:44.636651    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="088b5c13f4173163a83e54a2400ad76c8fc610d69a6bac92d013dceca14f4ec0"
Nov 03 21:08:46.220385 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:46.220326    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:46.339099 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:46.339068    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"]
Nov 03 21:08:46.339329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:46.339307    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" containerID="cri-o://e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed" gracePeriod=30
Nov 03 21:08:49.065718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:49.065676    5318 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:08:49.066206 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:49.065738    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:08:50.524321 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:50.524270    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:51.619187 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.619110    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 21:08:51.687233 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.687202    5318 generic.go:356] "Generic (PLEG): container finished" podID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerID="e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed" exitCode=0
Nov 03 21:08:51.687416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.687250    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"
Nov 03 21:08:51.687416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.687295    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" event={"ID":"f9b15112-2cf7-4c6b-9900-8dd57239041b","Type":"ContainerDied","Data":"e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed"}
Nov 03 21:08:51.687416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.687386    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7" event={"ID":"f9b15112-2cf7-4c6b-9900-8dd57239041b","Type":"ContainerDied","Data":"be97bbd5356420a2fa59ee3b8aee348788bd995f89554b6c20a32c52bcbc6a7e"}
Nov 03 21:08:51.687540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.687417    5318 scope.go:122] "RemoveContainer" containerID="e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed"
Nov 03 21:08:51.697217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.697197    5318 scope.go:122] "RemoveContainer" containerID="e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed"
Nov 03 21:08:51.697602 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:51.697574    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed\": container with ID starting with e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed not found: ID does not exist" containerID="e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed"
Nov 03 21:08:51.697675 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.697617    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed"} err="failed to get container status \"e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed\": rpc error: code = NotFound desc = could not find container \"e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed\": container with ID starting with e80570da26ab432ee17bf0d2b47fd6b4c8c2e04543de52ff72a3b344042b09ed not found: ID does not exist"
Nov 03 21:08:51.743948 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.743916    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744017    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744061    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744079    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744106    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744151    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744174    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744193    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:51.744399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744243    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744301    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo\") pod \"f9b15112-2cf7-4c6b-9900-8dd57239041b\" (UID: \"f9b15112-2cf7-4c6b-9900-8dd57239041b\") "
Nov 03 21:08:51.744399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744332    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:51.744721 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744402    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:51.744721 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744637    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:51.744721 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744646    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:08:51.745038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744926    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.745038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744951    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.745038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744968    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.745038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744982    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.745434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.744995    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b15112-2cf7-4c6b-9900-8dd57239041b-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.746583 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.746557    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "kube-api-access-phx5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:51.746677 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.746643    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:08:51.746873 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.746850    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:08:51.746944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.746903    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo" pod "f9b15112-2cf7-4c6b-9900-8dd57239041b" (UID: "f9b15112-2cf7-4c6b-9900-8dd57239041b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:08:51.846527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.846491    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.846527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.846526    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.846736 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.846541    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phx5f\" (UniqueName: \"kubernetes.io/projected/f9b15112-2cf7-4c6b-9900-8dd57239041b-kube-api-access-phx5f\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:51.846736 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:51.846555    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b15112-2cf7-4c6b-9900-8dd57239041b-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:08:52.022928 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:52.022899    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:52.150485 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:52.150442    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"]
Nov 03 21:08:52.156009 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:52.155964    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-w64l7"]
Nov 03 21:08:52.230110 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:52.230071    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:52.512525 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:52.512495    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:52.724370 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:52.724322    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:53.148209 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:53.148180    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:53.255427 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:53.255394    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" path="/var/lib/kubelet/pods/f9b15112-2cf7-4c6b-9900-8dd57239041b/volumes"
Nov 03 21:08:54.315646 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:54.315615    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:55.935327 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:55.935293    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:08:56.659325 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:56.659294    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:56.888183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:56.888155    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:57.357788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.357757    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"]
Nov 03 21:08:57.358287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.358274    5318 cpu_manager.go:414] "containerMap: removing container" podUID="a5773541-4239-4aa9-b981-3939bcc5dbcc" containerName="revoke-keys"
Nov 03 21:08:57.358356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.358288    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5773541-4239-4aa9-b981-3939bcc5dbcc" containerName="revoke-keys"
Nov 03 21:08:57.358356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.358306    5318 cpu_manager.go:414] "containerMap: removing container" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy"
Nov 03 21:08:57.358356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.358311    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy"
Nov 03 21:08:57.358463 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.358436    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5773541-4239-4aa9-b981-3939bcc5dbcc" containerName="revoke-keys"
Nov 03 21:08:57.358463 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.358447    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b15112-2cf7-4c6b-9900-8dd57239041b" containerName="istio-proxy"
Nov 03 21:08:57.363443 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.363420    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.425052 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.425015    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67gv\" (UniqueName: \"kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.425204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.425190    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.425268 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.425228    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.425323 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.425287    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.463184 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.463150    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"]
Nov 03 21:08:57.476155 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.476129    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8afb98-serving-cert\""
Nov 03 21:08:57.526978 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.526935    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.527123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.526994    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.527123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.527055    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.527226 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:57.527140    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-serving-cert: secret "maas-api-e2e-api-a-8afb98-serving-cert" not found
Nov 03 21:08:57.527226 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.527177    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67gv\" (UniqueName: \"kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.527226 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:08:57.527208    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls podName:a174b6b5-a6d0-4f7e-a362-552b9c76a47e nodeName:}" failed. No retries permitted until 2026-11-03 21:08:58.027190654 +0000 UTC m=+2331.006747476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls") pod "maas-api-e2e-api-a-8afb98-766c65c99-qljwl" (UID: "a174b6b5-a6d0-4f7e-a362-552b9c76a47e") : secret "maas-api-e2e-api-a-8afb98-serving-cert" not found
Nov 03 21:08:57.527749 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.527728    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.529472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.529450    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.584839 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.584809    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67gv\" (UniqueName: \"kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:57.611674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.611586    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:57.966853 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:57.966768    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:58.033146 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:58.033114    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:58.035488 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:58.035468    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\") pod \"maas-api-e2e-api-a-8afb98-766c65c99-qljwl\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:58.279869 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:58.279766    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:58.465021 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:58.464991    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:58.676687 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:58.676644    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"]
Nov 03 21:08:58.679326 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:08:58.679298    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda174b6b5_a6d0_4f7e_a362_552b9c76a47e.slice/crio-91c119977860bbd62b4567d204a9ca3c77e579e9992556c234b7b5dcc822de63 WatchSource:0}: Error finding container 91c119977860bbd62b4567d204a9ca3c77e579e9992556c234b7b5dcc822de63: Status 404 returned error can't find the container with id 91c119977860bbd62b4567d204a9ca3c77e579e9992556c234b7b5dcc822de63
Nov 03 21:08:58.736030 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:58.735998    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl" event={"ID":"a174b6b5-a6d0-4f7e-a362-552b9c76a47e","Type":"ContainerStarted","Data":"91c119977860bbd62b4567d204a9ca3c77e579e9992556c234b7b5dcc822de63"}
Nov 03 21:08:59.745027 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:59.744991    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl" event={"ID":"a174b6b5-a6d0-4f7e-a362-552b9c76a47e","Type":"ContainerStarted","Data":"4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34"}
Nov 03 21:08:59.745498 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:59.745171    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:08:59.856391 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:08:59.856312    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl" podStartSLOduration=1.986642617 podStartE2EDuration="2.856297606s" totalImagesPullingTime="869.654989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:59.854930692 +0000 UTC m=+2332.834487532" watchObservedRunningTime="2026-11-03 21:08:59.856297606 +0000 UTC m=+2332.835854459"
Nov 03 21:09:00.220826 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:00.220722    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:00.470372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:00.470306    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:02.799855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:02.799820    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:05.757179 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:05.757152    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:09:07.513951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:07.513912    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:09:07.679760 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:07.679710    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:07.903321 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:07.903251    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:13.159868 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:13.159833    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:18.419543 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:18.419448    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:23.399685 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:23.399639    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:25.932812 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:09:25.932775    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:09:26.528184 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:26.528155    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:27.779407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:27.779370    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:29.447695 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:29.447639    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:29.656777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:29.656746    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:29.864480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:29.864449    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:30.350126 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:30.350096    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:34.957017 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:34.956985    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:35.176018 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:35.175980    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:35.523175 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:35.523145    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:37.063986 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:37.063958    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:37.280304 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:37.280260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:37.491450 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:37.491417    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:37.856605 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:37.856574    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:40.752164 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:40.752131    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:41.450684 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:41.450649    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:42.609801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:42.609767    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:42.775210 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:42.775169    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"]
Nov 03 21:09:42.998407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:42.998269    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:43.257093 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:43.257017    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:43.536167 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:43.536069    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:45.487917 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:45.487875    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:45.831951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:45.831849    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:46.676796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:46.676771    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:46.881033 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:46.880989    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:47.238425 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:47.238373    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:47.486371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:47.486298    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:47.712177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:47.712116    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:47.782050 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:47.782020    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"]
Nov 03 21:09:47.828451 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:47.828416    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3d2008-st5bx"]
Nov 03 21:09:48.218292 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:48.218260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:48.436455 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:48.436418    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:48.792782 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:48.792749    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:49.081052 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:49.080947    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:49.255785 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:49.255748    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f522098-dd68-4c8e-86d8-fc34725d54af" path="/var/lib/kubelet/pods/5f522098-dd68-4c8e-86d8-fc34725d54af/volumes"
Nov 03 21:09:49.307977 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:49.307907    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:50.136243 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:50.136204    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"]
Nov 03 21:09:50.552935 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:50.552825    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:50.759034 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:50.758999    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:51.658497 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:51.658461    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:52.651216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:52.651189    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:53.082650 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.082578    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"]
Nov 03 21:09:53.087440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.087414    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.136645 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.136622    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:09:53.182184 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.182149    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:09:53.186407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.186376    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"]
Nov 03 21:09:53.214386 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.214362    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.214508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.214432    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.214508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.214486    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fnc\" (UniqueName: \"kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.317405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.316750    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.317405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.316842    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.317405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.316884    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fnc\" (UniqueName: \"kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.318364 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.318110    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.326389 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.326166    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.374463 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.374430    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fnc\" (UniqueName: \"kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc\") pod \"maas-api-revoke-sub-04e32281-w5zjt\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.408226 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.408197    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:53.780557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:53.780524    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"]
Nov 03 21:09:53.780966 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:09:53.780932    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12884503_f8f9_4720_96f7_94d3345939fb.slice/crio-f2ad62ec5e332f4866be3110f98f9ea220b5fd1378d97492def2d5aa9e0d7070 WatchSource:0}: Error finding container f2ad62ec5e332f4866be3110f98f9ea220b5fd1378d97492def2d5aa9e0d7070: Status 404 returned error can't find the container with id f2ad62ec5e332f4866be3110f98f9ea220b5fd1378d97492def2d5aa9e0d7070
Nov 03 21:09:54.137903 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.137871    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt" event={"ID":"12884503-f8f9-4720-96f7-94d3345939fb","Type":"ContainerStarted","Data":"8c65f6f4661f9104e449ca243f407b87076841e1c55faf9dae91904c30ffd9de"}
Nov 03 21:09:54.138251 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.137934    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt" event={"ID":"12884503-f8f9-4720-96f7-94d3345939fb","Type":"ContainerStarted","Data":"f2ad62ec5e332f4866be3110f98f9ea220b5fd1378d97492def2d5aa9e0d7070"}
Nov 03 21:09:54.173155 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.173125    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:54.246298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.246227    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt" podStartSLOduration=1.246209038 podStartE2EDuration="1.246209038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:54.245783513 +0000 UTC m=+2387.225340366" watchObservedRunningTime="2026-11-03 21:09:54.246209038 +0000 UTC m=+2387.225765879"
Nov 03 21:09:54.396603 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.396557    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:54.512792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.512758    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"]
Nov 03 21:09:54.517665 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.517638    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.573719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.573694    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f1855-dockercfg-dhm6w\""
Nov 03 21:09:54.608138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.608111    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f1855\""
Nov 03 21:09:54.615392 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.613277    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"]
Nov 03 21:09:54.641201 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.641164    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.641396 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.641296    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgtr\" (UniqueName: \"kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr\") pod \"payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.641739 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.641713    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:54.668177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.668093    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"]
Nov 03 21:09:54.677005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.676969    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:54.750123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.750085    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:54.750435 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.750411    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.750638 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.750622    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptml\" (UniqueName: \"kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml\") pod \"payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:54.750782 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.750768    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgtr\" (UniqueName: \"kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr\") pod \"payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.752136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.752113    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.758585 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.758557    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"]
Nov 03 21:09:54.852387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.847928    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"]
Nov 03 21:09:54.852387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.848167    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy" containerID="cri-o://66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462" gracePeriod=30
Nov 03 21:09:54.852387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.851035    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgtr\" (UniqueName: \"kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr\") pod \"payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:54.852387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.851980    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptml\" (UniqueName: \"kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml\") pod \"payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:54.852387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.852230    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:54.856391 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.853213    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:54.876418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.876387    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:54.917116 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:54.917084    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptml\" (UniqueName: \"kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml\") pod \"payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:55.002733 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.002643    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:55.122047 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.121968    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:55.138431 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.136825    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:55.139197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.139019    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerName="payload-processing" containerID="cri-o://1afd293af81833d9e01e2400fa8ef272479cdac9afa556d5673232eccb70418a" gracePeriod=25
Nov 03 21:09:55.158253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.157693    5318 generic.go:356] "Generic (PLEG): container finished" podID="12884503-f8f9-4720-96f7-94d3345939fb" containerID="8c65f6f4661f9104e449ca243f407b87076841e1c55faf9dae91904c30ffd9de" exitCode=0
Nov 03 21:09:55.158253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.157878    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt" event={"ID":"12884503-f8f9-4720-96f7-94d3345939fb","Type":"ContainerDied","Data":"8c65f6f4661f9104e449ca243f407b87076841e1c55faf9dae91904c30ffd9de"}
Nov 03 21:09:55.381272 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.381243    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"]
Nov 03 21:09:55.384043 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:09:55.384012    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65d8223_fbbd_45fe_a686_8188a262d7d9.slice/crio-b74568d9bc7629de61d8b2e6050f7d2826716f666820c915a520c46912e17cab WatchSource:0}: Error finding container b74568d9bc7629de61d8b2e6050f7d2826716f666820c915a520c46912e17cab: Status 404 returned error can't find the container with id b74568d9bc7629de61d8b2e6050f7d2826716f666820c915a520c46912e17cab
Nov 03 21:09:55.556487 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:55.556463    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"]
Nov 03 21:09:55.557918 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:09:55.557885    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa3a3b6_6150_4cb7_9934_d2bee90b9763.slice/crio-92efe6ac61d8d6525c3202b7c7784f10d4de2942b0a1d6ede267a876df37bfcf WatchSource:0}: Error finding container 92efe6ac61d8d6525c3202b7c7784f10d4de2942b0a1d6ede267a876df37bfcf: Status 404 returned error can't find the container with id 92efe6ac61d8d6525c3202b7c7784f10d4de2942b0a1d6ede267a876df37bfcf
Nov 03 21:09:55.934777 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:09:55.934701    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:09:56.166145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.166107    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" event={"ID":"bfa3a3b6-6150-4cb7-9934-d2bee90b9763","Type":"ContainerStarted","Data":"efe5e2e3e0a5f975f6e7f86721309d98206801fe61b6b836d6a27a305651926c"}
Nov 03 21:09:56.166659 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.166168    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" event={"ID":"bfa3a3b6-6150-4cb7-9934-d2bee90b9763","Type":"ContainerStarted","Data":"92efe6ac61d8d6525c3202b7c7784f10d4de2942b0a1d6ede267a876df37bfcf"}
Nov 03 21:09:56.166659 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.166196    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:09:56.167952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.167921    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" event={"ID":"e65d8223-fbbd-45fe-a686-8188a262d7d9","Type":"ContainerStarted","Data":"d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496"}
Nov 03 21:09:56.168097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.167969    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" event={"ID":"e65d8223-fbbd-45fe-a686-8188a262d7d9","Type":"ContainerStarted","Data":"b74568d9bc7629de61d8b2e6050f7d2826716f666820c915a520c46912e17cab"}
Nov 03 21:09:56.168097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.167995    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:09:56.299092 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.299061    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:56.301867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.301824    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" podStartSLOduration=2.301810808 podStartE2EDuration="2.301810808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:56.29841722 +0000 UTC m=+2389.277974061" watchObservedRunningTime="2026-11-03 21:09:56.301810808 +0000 UTC m=+2389.281367648"
Nov 03 21:09:56.372074 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.372036    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca\") pod \"12884503-f8f9-4720-96f7-94d3345939fb\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") "
Nov 03 21:09:56.372253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.372104    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca\") pod \"12884503-f8f9-4720-96f7-94d3345939fb\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") "
Nov 03 21:09:56.372253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.372191    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc\" (UniqueName: \"kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc\") pod \"12884503-f8f9-4720-96f7-94d3345939fb\" (UID: \"12884503-f8f9-4720-96f7-94d3345939fb\") "
Nov 03 21:09:56.372469 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.372446    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca" pod "12884503-f8f9-4720-96f7-94d3345939fb" (UID: "12884503-f8f9-4720-96f7-94d3345939fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:56.372922 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.372895    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:56.374448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.374422    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc" pod "12884503-f8f9-4720-96f7-94d3345939fb" (UID: "12884503-f8f9-4720-96f7-94d3345939fb"). InnerVolumeSpecName "kube-api-access-s5fnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:56.374568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.374492    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca" pod "12884503-f8f9-4720-96f7-94d3345939fb" (UID: "12884503-f8f9-4720-96f7-94d3345939fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:56.450434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.450370    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" podStartSLOduration=2.450329405 podStartE2EDuration="2.450329405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:56.447554713 +0000 UTC m=+2389.427111632" watchObservedRunningTime="2026-11-03 21:09:56.450329405 +0000 UTC m=+2389.429886249"
Nov 03 21:09:56.473919 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.473831    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12884503-f8f9-4720-96f7-94d3345939fb-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:56.473919 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:56.473867    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5fnc\" (UniqueName: \"kubernetes.io/projected/12884503-f8f9-4720-96f7-94d3345939fb-kube-api-access-s5fnc\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:57.180394 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:57.180362    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"
Nov 03 21:09:57.180394 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:57.180386    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt" event={"ID":"12884503-f8f9-4720-96f7-94d3345939fb","Type":"ContainerDied","Data":"f2ad62ec5e332f4866be3110f98f9ea220b5fd1378d97492def2d5aa9e0d7070"}
Nov 03 21:09:57.180944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:57.180466    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ad62ec5e332f4866be3110f98f9ea220b5fd1378d97492def2d5aa9e0d7070"
Nov 03 21:09:57.640067 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:57.640029    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:09:57.800186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:57.800145    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" containerID="cri-o://574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a" gracePeriod=45
Nov 03 21:09:57.950687 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:57.950648    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused"
Nov 03 21:09:58.065120 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.065093    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 21:09:58.191444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.191329    5318 generic.go:356] "Generic (PLEG): container finished" podID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerID="574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a" exitCode=0
Nov 03 21:09:58.191444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.191412    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"
Nov 03 21:09:58.192006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.191592    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" event={"ID":"b4526ab7-af1a-43eb-be89-429e62e1c5f9","Type":"ContainerDied","Data":"574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a"}
Nov 03 21:09:58.192006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.191627    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7" event={"ID":"b4526ab7-af1a-43eb-be89-429e62e1c5f9","Type":"ContainerDied","Data":"40add25b7455fe96ae13048d4988b0ea35ef6daaad4fe6a9d29367b97ec0fcea"}
Nov 03 21:09:58.192006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.191650    5318 scope.go:122] "RemoveContainer" containerID="574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a"
Nov 03 21:09:58.194499 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.194462    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache\") pod \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") "
Nov 03 21:09:58.194616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.194535    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs\") pod \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") "
Nov 03 21:09:58.194682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.194620    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir\") pod \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") "
Nov 03 21:09:58.194682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.194674    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4\" (UniqueName: \"kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4\") pod \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") "
Nov 03 21:09:58.194793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.194712    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home\") pod \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") "
Nov 03 21:09:58.194793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.194779    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm\") pod \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\" (UID: \"b4526ab7-af1a-43eb-be89-429e62e1c5f9\") "
Nov 03 21:09:58.195780 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.195755    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir" pod "b4526ab7-af1a-43eb-be89-429e62e1c5f9" (UID: "b4526ab7-af1a-43eb-be89-429e62e1c5f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.196199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.196172    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home" pod "b4526ab7-af1a-43eb-be89-429e62e1c5f9" (UID: "b4526ab7-af1a-43eb-be89-429e62e1c5f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.196622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.196598    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache" pod "b4526ab7-af1a-43eb-be89-429e62e1c5f9" (UID: "b4526ab7-af1a-43eb-be89-429e62e1c5f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.198635 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.198611    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm" pod "b4526ab7-af1a-43eb-be89-429e62e1c5f9" (UID: "b4526ab7-af1a-43eb-be89-429e62e1c5f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:58.198874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.198853    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs" pod "b4526ab7-af1a-43eb-be89-429e62e1c5f9" (UID: "b4526ab7-af1a-43eb-be89-429e62e1c5f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:58.201404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.199002    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4" pod "b4526ab7-af1a-43eb-be89-429e62e1c5f9" (UID: "b4526ab7-af1a-43eb-be89-429e62e1c5f9"). InnerVolumeSpecName "kube-api-access-v8dt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:58.207495 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.207475    5318 scope.go:122] "RemoveContainer" containerID="574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a"
Nov 03 21:09:58.207932 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:09:58.207837    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a\": container with ID starting with 574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a not found: ID does not exist" containerID="574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a"
Nov 03 21:09:58.207932 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.207875    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a"} err="failed to get container status \"574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a\": rpc error: code = NotFound desc = could not find container \"574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a\": container with ID starting with 574b1e01dd0d88593ab81d3c7b213957d140eafeb52b15744050387c4f9ebd9a not found: ID does not exist"
Nov 03 21:09:58.271631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.271598    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:58.296211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.296175    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.296211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.296211    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dt4\" (UniqueName: \"kubernetes.io/projected/b4526ab7-af1a-43eb-be89-429e62e1c5f9-kube-api-access-v8dt4\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.296211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.296227    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.296526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.296240    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.296526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.296253    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4526ab7-af1a-43eb-be89-429e62e1c5f9-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.296526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.296267    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4526ab7-af1a-43eb-be89-429e62e1c5f9-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:58.455556 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.452485    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"]
Nov 03 21:09:58.455556 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.452769    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" containerID="cri-o://55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a" gracePeriod=30
Nov 03 21:09:58.473770 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.473729    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"]
Nov 03 21:09:58.474164 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.474137    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" containerID="cri-o://cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5" gracePeriod=30
Nov 03 21:09:58.481543 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.481516    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"]
Nov 03 21:09:58.481760 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.481734    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy" containerID="cri-o://011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8" gracePeriod=30
Nov 03 21:09:58.521140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.521107    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:58.744769 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.744652    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:58.803585 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.803558    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"]
Nov 03 21:09:58.806568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:58.806537    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/premium-w5-2a5ec9-kserve-7bc7965bd-plhz7"]
Nov 03 21:09:59.264084 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:59.264052    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" path="/var/lib/kubelet/pods/b4526ab7-af1a-43eb-be89-429e62e1c5f9/volumes"
Nov 03 21:09:59.363574 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:59.363539    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:59.578729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:59.578653    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:09:59.801125 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:09:59.801096    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:00.122994 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.122938    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 21:10:00.156258 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.156231    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:00.205330 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.205294    5318 generic.go:356] "Generic (PLEG): container finished" podID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerID="66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462" exitCode=0
Nov 03 21:10:00.205527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.205372    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"
Nov 03 21:10:00.205527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.205382    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" event={"ID":"c6b6e8ce-6f19-479a-8ab4-38368e5b878c","Type":"ContainerDied","Data":"66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462"}
Nov 03 21:10:00.205527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.205441    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x" event={"ID":"c6b6e8ce-6f19-479a-8ab4-38368e5b878c","Type":"ContainerDied","Data":"5a3d72fb343027c23fc45f78627028afa1ba809a2eb7d10dfe5ac4b06075a5e2"}
Nov 03 21:10:00.205527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.205469    5318 scope.go:122] "RemoveContainer" containerID="66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462"
Nov 03 21:10:00.215645 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.215626    5318 scope.go:122] "RemoveContainer" containerID="66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462"
Nov 03 21:10:00.215968 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:00.215943    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462\": container with ID starting with 66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462 not found: ID does not exist" containerID="66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462"
Nov 03 21:10:00.216060 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.215978    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462"} err="failed to get container status \"66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462\": rpc error: code = NotFound desc = could not find container \"66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462\": container with ID starting with 66f50b726784fe8081345e61f60ecebba9a84b12a2e133023e26ec3529e74462 not found: ID does not exist"
Nov 03 21:10:00.230039 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230015    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230154 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230099    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230154 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230150    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230213    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230242    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230401 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230288    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230396    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230507 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230492    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.230645 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.230630    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket\") pod \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\" (UID: \"c6b6e8ce-6f19-479a-8ab4-38368e5b878c\") "
Nov 03 21:10:00.231318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.231274    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:00.231469 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.231385    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:00.231469 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.231407    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:00.231574 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.231525    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:00.231813 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.231791    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:00.232509 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.232486    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "kube-api-access-5dt8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:00.233588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.233562    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:00.234063 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.234035    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:00.234153 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.234069    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token" pod "c6b6e8ce-6f19-479a-8ab4-38368e5b878c" (UID: "c6b6e8ce-6f19-479a-8ab4-38368e5b878c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:00.332866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332831    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.332866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332861    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.332866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332870    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.333479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332879    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.333479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332889    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dt8d\" (UniqueName: \"kubernetes.io/projected/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-kube-api-access-5dt8d\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.333479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332898    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.333479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332906    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.333479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332914    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.333479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.332922    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b6e8ce-6f19-479a-8ab4-38368e5b878c-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:00.620588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.620557    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"]
Nov 03 21:10:00.628916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:00.628878    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-4557x"]
Nov 03 21:10:01.255895 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:01.255861    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" path="/var/lib/kubelet/pods/c6b6e8ce-6f19-479a-8ab4-38368e5b878c/volumes"
Nov 03 21:10:01.300229 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:01.300201    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:02.931843 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:02.931803    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:02.932327 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:02.931879    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:03.289288 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.289212    5318 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:10:03.289448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.289277    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:10:03.760471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.760444    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:10:03.829143 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.829117    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:10:03.832855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.832832    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:10:03.876775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.876743    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.876947 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.876918    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.877033 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.876965    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.877087 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877050    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.877087 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877080    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.877198 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877116    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.877198 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877180    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877223    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877237    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877257    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token\") pod \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\" (UID: \"40fcd3de-0656-4b21-a46e-3e76fd5fc6be\") "
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877498    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877857    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.877879    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.878111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.878040    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:03.878532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.878128    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.878532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.878178    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.879306 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.879286    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.879544 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.879524    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:03.879544 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.879534    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "kube-api-access-jdvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.879730 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.879712    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token" pod "40fcd3de-0656-4b21-a46e-3e76fd5fc6be" (UID: "40fcd3de-0656-4b21-a46e-3e76fd5fc6be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.978693 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.978657    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.978715    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.978733    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.978775    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.978816    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.978935    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979000    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979049    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979119    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.979208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979115    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979224    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979369    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979386    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979513    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979529    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979563    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979604    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979642    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.979719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979672    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.980204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979792    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy\") pod \"60e65ae6-cd43-447e-9665-af5734e5413e\" (UID: \"60e65ae6-cd43-447e-9665-af5734e5413e\") "
Nov 03 21:10:03.980204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979842    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.980204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979889    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.980204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979919    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q\") pod \"2776d738-536e-40aa-b7aa-8f96546f1758\" (UID: \"2776d738-536e-40aa-b7aa-8f96546f1758\") "
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980505    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60e65ae6-cd43-447e-9665-af5734e5413e-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980532    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2776d738-536e-40aa-b7aa-8f96546f1758-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980550    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvzf\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-kube-api-access-jdvzf\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980566    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980580    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980596    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980610    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980626    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980640    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980654    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980668    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.980682    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40fcd3de-0656-4b21-a46e-3e76fd5fc6be-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979793    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.979953    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.981327    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.981371    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.981383    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.981604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.981574    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.982926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.982187    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:03.982926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.982474    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.982926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.982509    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "kube-api-access-m2mqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.982926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.982565    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:03.982926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.982709    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:10:03.983254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.983237    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy" pod "60e65ae6-cd43-447e-9665-af5734e5413e" (UID: "60e65ae6-cd43-447e-9665-af5734e5413e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:10:03.983875 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:03.983852    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q" pod "2776d738-536e-40aa-b7aa-8f96546f1758" (UID: "2776d738-536e-40aa-b7aa-8f96546f1758"). InnerVolumeSpecName "kube-api-access-s9n2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081795    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081829    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60e65ae6-cd43-447e-9665-af5734e5413e-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081840    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mqg\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-kube-api-access-m2mqg\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081850    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2776d738-536e-40aa-b7aa-8f96546f1758-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081858    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081867    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60e65ae6-cd43-447e-9665-af5734e5413e-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081875    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.081886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081884    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.082449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081895    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60e65ae6-cd43-447e-9665-af5734e5413e-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.082449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081907    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.082449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081916    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2776d738-536e-40aa-b7aa-8f96546f1758-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.082449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081924    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9n2q\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-kube-api-access-s9n2q\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.082449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.081932    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2776d738-536e-40aa-b7aa-8f96546f1758-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:04.234863 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.234830    5318 generic.go:356] "Generic (PLEG): container finished" podID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerID="cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5" exitCode=0
Nov 03 21:10:04.235037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.234899    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"
Nov 03 21:10:04.235037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.234913    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" event={"ID":"40fcd3de-0656-4b21-a46e-3e76fd5fc6be","Type":"ContainerDied","Data":"cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5"}
Nov 03 21:10:04.235037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.235008    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn" event={"ID":"40fcd3de-0656-4b21-a46e-3e76fd5fc6be","Type":"ContainerDied","Data":"3fab6d8deb026e49f54213025de6d325ebb9b65b3acfcc880c8adc507ee4a575"}
Nov 03 21:10:04.235237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.235044    5318 scope.go:122] "RemoveContainer" containerID="cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5"
Nov 03 21:10:04.237487 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.237459    5318 generic.go:356] "Generic (PLEG): container finished" podID="2776d738-536e-40aa-b7aa-8f96546f1758" containerID="55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a" exitCode=0
Nov 03 21:10:04.237592 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.237537    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"
Nov 03 21:10:04.237658 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.237580    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" event={"ID":"2776d738-536e-40aa-b7aa-8f96546f1758","Type":"ContainerDied","Data":"55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a"}
Nov 03 21:10:04.237658 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.237626    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7" event={"ID":"2776d738-536e-40aa-b7aa-8f96546f1758","Type":"ContainerDied","Data":"315367e33fa3eced22db597856066484b4842c4a2e3480560421b727b882f50a"}
Nov 03 21:10:04.239122 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.239099    5318 generic.go:356] "Generic (PLEG): container finished" podID="60e65ae6-cd43-447e-9665-af5734e5413e" containerID="011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8" exitCode=0
Nov 03 21:10:04.239246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.239127    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" event={"ID":"60e65ae6-cd43-447e-9665-af5734e5413e","Type":"ContainerDied","Data":"011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8"}
Nov 03 21:10:04.239365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.239276    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l" event={"ID":"60e65ae6-cd43-447e-9665-af5734e5413e","Type":"ContainerDied","Data":"72c30b9ad72bea659af14f0fc303acc685d107cc4a2f988cd4b5b0cded22481b"}
Nov 03 21:10:04.239365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.239169    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"
Nov 03 21:10:04.247607 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.247588    5318 scope.go:122] "RemoveContainer" containerID="cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5"
Nov 03 21:10:04.247975 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:04.247940    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5\": container with ID starting with cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5 not found: ID does not exist" containerID="cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5"
Nov 03 21:10:04.248140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.247989    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5"} err="failed to get container status \"cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5\": rpc error: code = NotFound desc = could not find container \"cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5\": container with ID starting with cbb761d5b5f6589cc6512253ec492a786c779e0c4fd14be6dec02e059efdc3d5 not found: ID does not exist"
Nov 03 21:10:04.248140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.248013    5318 scope.go:122] "RemoveContainer" containerID="55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a"
Nov 03 21:10:04.258143 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.258129    5318 scope.go:122] "RemoveContainer" containerID="55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a"
Nov 03 21:10:04.258425 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:04.258407    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a\": container with ID starting with 55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a not found: ID does not exist" containerID="55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a"
Nov 03 21:10:04.258475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.258431    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a"} err="failed to get container status \"55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a\": rpc error: code = NotFound desc = could not find container \"55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a\": container with ID starting with 55bf1f59ab73c8079495e7e57ee4a96ae164eadd37b67b4785fe703a0fe5375a not found: ID does not exist"
Nov 03 21:10:04.258475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.258446    5318 scope.go:122] "RemoveContainer" containerID="011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8"
Nov 03 21:10:04.267410 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.267395    5318 scope.go:122] "RemoveContainer" containerID="011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8"
Nov 03 21:10:04.267680 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:04.267661    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8\": container with ID starting with 011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8 not found: ID does not exist" containerID="011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8"
Nov 03 21:10:04.267771 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.267685    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8"} err="failed to get container status \"011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8\": rpc error: code = NotFound desc = could not find container \"011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8\": container with ID starting with 011277c64bee4d1c4be93987351c4a07ffb77936b3e4296721d3483354801cd8 not found: ID does not exist"
Nov 03 21:10:04.393858 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.393832    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"]
Nov 03 21:10:04.399438 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.399417    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-d84zn"]
Nov 03 21:10:04.501322 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.501288    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"]
Nov 03 21:10:04.506130 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.506104    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-xrl6l"]
Nov 03 21:10:04.609405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.609374    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"]
Nov 03 21:10:04.613998 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:04.613976    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-vtsw7"]
Nov 03 21:10:05.254303 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:05.254270    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" path="/var/lib/kubelet/pods/2776d738-536e-40aa-b7aa-8f96546f1758/volumes"
Nov 03 21:10:05.254919 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:05.254903    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" path="/var/lib/kubelet/pods/40fcd3de-0656-4b21-a46e-3e76fd5fc6be/volumes"
Nov 03 21:10:05.255323 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:05.255311    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" path="/var/lib/kubelet/pods/60e65ae6-cd43-447e-9665-af5734e5413e/volumes"
Nov 03 21:10:05.948527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:05.948497    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:07.183793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:07.183766    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:10:07.184249 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:07.184170    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:10:07.639496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:07.639456    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:07.668793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:07.668765    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:10:07.673125 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:07.673105    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:10:11.321320 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:11.321289    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:12.036888 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:12.036849    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:12.668388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:12.668351    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:13.103187 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:13.103154    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:14.709621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:14.709589    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:15.313999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.313962    5318 generic.go:356] "Generic (PLEG): container finished" podID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerID="1afd293af81833d9e01e2400fa8ef272479cdac9afa556d5673232eccb70418a" exitCode=0
Nov 03 21:10:15.314161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.314037    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" event={"ID":"a88ef88e-715e-4a3f-863f-25319fc0a555","Type":"ContainerDied","Data":"1afd293af81833d9e01e2400fa8ef272479cdac9afa556d5673232eccb70418a"}
Nov 03 21:10:15.434366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.434323    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 21:10:15.522648 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.522621    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume\") pod \"a88ef88e-715e-4a3f-863f-25319fc0a555\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") "
Nov 03 21:10:15.522796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.522692    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq\" (UniqueName: \"kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq\") pod \"a88ef88e-715e-4a3f-863f-25319fc0a555\" (UID: \"a88ef88e-715e-4a3f-863f-25319fc0a555\") "
Nov 03 21:10:15.522991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.522968    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume" pod "a88ef88e-715e-4a3f-863f-25319fc0a555" (UID: "a88ef88e-715e-4a3f-863f-25319fc0a555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:15.523147 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.523135    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88ef88e-715e-4a3f-863f-25319fc0a555-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:15.524676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.524649    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq" pod "a88ef88e-715e-4a3f-863f-25319fc0a555" (UID: "a88ef88e-715e-4a3f-863f-25319fc0a555"). InnerVolumeSpecName "kube-api-access-gr7gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:15.624395 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:15.624357    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7gq\" (UniqueName: \"kubernetes.io/projected/a88ef88e-715e-4a3f-863f-25319fc0a555-kube-api-access-gr7gq\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:16.321568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.321523    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp" event={"ID":"a88ef88e-715e-4a3f-863f-25319fc0a555","Type":"ContainerDied","Data":"35d568b9c9ee1eccb0fde30a136a724c9326376ffe389ebf6c83f2595d376845"}
Nov 03 21:10:16.321568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.321549    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"
Nov 03 21:10:16.322106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.321593    5318 scope.go:122] "RemoveContainer" containerID="1afd293af81833d9e01e2400fa8ef272479cdac9afa556d5673232eccb70418a"
Nov 03 21:10:16.465682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.465652    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:16.479612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.479584    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"]
Nov 03 21:10:16.524407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.524380    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2a5ec9-5f66c56d59-rgjmp"]
Nov 03 21:10:16.758377 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.758321    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:16.977420 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:16.977392    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:17.257141 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:17.257110    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" path="/var/lib/kubelet/pods/a88ef88e-715e-4a3f-863f-25319fc0a555/volumes"
Nov 03 21:10:18.372361 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:18.372322    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:10:19.969403 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:19.969372    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:20.186365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.181891    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.355893    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"]
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358359    5318 cpu_manager.go:414] "containerMap: removing container" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358383    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358403    5318 cpu_manager.go:414] "containerMap: removing container" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358416    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358453    5318 cpu_manager.go:414] "containerMap: removing container" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358463    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358482    5318 cpu_manager.go:414] "containerMap: removing container" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358490    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358517    5318 cpu_manager.go:414] "containerMap: removing container" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerName="payload-processing"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358525    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerName="payload-processing"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358550    5318 cpu_manager.go:414] "containerMap: removing container" podUID="12884503-f8f9-4720-96f7-94d3345939fb" containerName="revoke-keys"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358557    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12884503-f8f9-4720-96f7-94d3345939fb" containerName="revoke-keys"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358578    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.358585    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359129    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88ef88e-715e-4a3f-863f-25319fc0a555" containerName="payload-processing"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359163    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="12884503-f8f9-4720-96f7-94d3345939fb" containerName="revoke-keys"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359174    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b6e8ce-6f19-479a-8ab4-38368e5b878c" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359203    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4526ab7-af1a-43eb-be89-429e62e1c5f9" containerName="main"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359216    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e65ae6-cd43-447e-9665-af5734e5413e" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359234    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fcd3de-0656-4b21-a46e-3e76fd5fc6be" containerName="istio-proxy"
Nov 03 21:10:20.360441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.359254    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="2776d738-536e-40aa-b7aa-8f96546f1758" containerName="istio-proxy"
Nov 03 21:10:20.384605 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.384571    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.445606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.445574    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cfcd54\""
Nov 03 21:10:20.445983 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.445961    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cfcd54-dockercfg-s42fn\""
Nov 03 21:10:20.450982 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.450695    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"]
Nov 03 21:10:20.545380 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.526106    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq\") pod \"payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.547564 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.547536    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.570253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.570210    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"]
Nov 03 21:10:20.642413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.628499    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.651929 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.651897    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"]
Nov 03 21:10:20.692094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.691971    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq\") pod \"payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.693230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.693205    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.694987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.694967    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.762469 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.762434    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq\") pod \"payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:20.794959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.794926    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.795434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.795399    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697vx\" (UniqueName: \"kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx\") pod \"payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.912316 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.912227    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697vx\" (UniqueName: \"kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx\") pod \"payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.912934 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.912909    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.921353 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.914143    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.981217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.981182    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-697vx\" (UniqueName: \"kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx\") pod \"payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:20.994155 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:20.994129    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:21.047765 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:21.047540    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:21.795586 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:21.795556    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"]
Nov 03 21:10:21.795830 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:21.795813    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"]
Nov 03 21:10:21.810073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:21.809284    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:22.021904 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.021667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:22.433528 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.433493    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl" event={"ID":"75dff506-ee70-4a03-97f6-b692b1fc9bd9","Type":"ContainerStarted","Data":"9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c"}
Nov 03 21:10:22.433719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.433541    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl" event={"ID":"75dff506-ee70-4a03-97f6-b692b1fc9bd9","Type":"ContainerStarted","Data":"aa5f6f6ccf267cf83253fa471765f21fbf7d3fb7ec7017364759260527f9d2d9"}
Nov 03 21:10:22.433719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.433576    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:22.435166 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.435141    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt" event={"ID":"42524b01-f10f-4c2d-8d40-47d31b9cf357","Type":"ContainerStarted","Data":"726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f"}
Nov 03 21:10:22.435301 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.435190    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:22.435301 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.435213    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt" event={"ID":"42524b01-f10f-4c2d-8d40-47d31b9cf357","Type":"ContainerStarted","Data":"54dea3b13d7072dac639ef28ee54c9ba2452e8f55ac26dff0e5c38bc114a2704"}
Nov 03 21:10:22.544537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.544492    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl" podStartSLOduration=2.544479143 podStartE2EDuration="2.544479143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:22.542135321 +0000 UTC m=+2415.521692161" watchObservedRunningTime="2026-11-03 21:10:22.544479143 +0000 UTC m=+2415.524036008"
Nov 03 21:10:22.653234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:22.653185    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt" podStartSLOduration=2.6531717710000002 podStartE2EDuration="2.653171771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:22.650626796 +0000 UTC m=+2415.630183638" watchObservedRunningTime="2026-11-03 21:10:22.653171771 +0000 UTC m=+2415.632728610"
Nov 03 21:10:23.191215 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:23.191182    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:25.933710 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:25.933674    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:10:26.890557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:26.890523    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:28.093054 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:28.093006    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:28.301688 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:28.301648    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:28.516766 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:28.516654    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:33.168599 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:33.168561    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:33.444501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:33.444428    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:10:33.444501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:33.444487    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:10:36.609915 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:36.609879    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:37.599326 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:37.599296    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:38.083742 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:38.083648    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:42.736689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.736650    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"]
Nov 03 21:10:42.741472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.741439    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.790204 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.790147    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f1855-serving-cert\""
Nov 03 21:10:42.834922 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.834883    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.835102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.834942    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.835155 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.835127    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.835207 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.835162    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5t2g\" (UniqueName: \"kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.839038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.839015    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"]
Nov 03 21:10:42.936784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.936748    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.936784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.936788    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5t2g\" (UniqueName: \"kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.937051 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.936838    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.937051 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.936873    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.937051 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:42.937007    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-serving-cert: secret "maas-api-e2e-inf-a-1f1855-serving-cert" not found
Nov 03 21:10:42.937217 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:42.937081    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls podName:b4f460c5-e0e9-4a59-a472-5ddedfc51e5f nodeName:}" failed. No retries permitted until 2026-11-03 21:10:43.437061078 +0000 UTC m=+2436.416617910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls") pod "maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" (UID: "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f") : secret "maas-api-e2e-inf-a-1f1855-serving-cert" not found
Nov 03 21:10:42.937560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.937536    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.939233 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.939213    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:42.997108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.996260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:42.998045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:42.998011    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5t2g\" (UniqueName: \"kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:43.220513 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:43.220480    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:43.442141 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:43.442106    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:43.444703 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:43.444673    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:43.654102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:43.654066    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:44.034350 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:44.034307    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"]
Nov 03 21:10:44.035910 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:10:44.035876    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f460c5_e0e9_4a59_a472_5ddedfc51e5f.slice/crio-8c45ff86e69d33664a46fa0e06d66e3b8ce3382a370c49dcb3449015fb36fc75 WatchSource:0}: Error finding container 8c45ff86e69d33664a46fa0e06d66e3b8ce3382a370c49dcb3449015fb36fc75: Status 404 returned error can't find the container with id 8c45ff86e69d33664a46fa0e06d66e3b8ce3382a370c49dcb3449015fb36fc75
Nov 03 21:10:44.570844 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:44.570807    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" event={"ID":"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f","Type":"ContainerStarted","Data":"8c45ff86e69d33664a46fa0e06d66e3b8ce3382a370c49dcb3449015fb36fc75"}
Nov 03 21:10:45.579870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:45.579780    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" event={"ID":"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f","Type":"ContainerStarted","Data":"37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec"}
Nov 03 21:10:45.579870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:45.579863    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:45.693737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:45.693680    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" podStartSLOduration=2.777571422 podStartE2EDuration="3.693666014s" totalImagesPullingTime="916.094592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:45.691614998 +0000 UTC m=+2438.671171837" watchObservedRunningTime="2026-11-03 21:10:45.693666014 +0000 UTC m=+2438.673222854"
Nov 03 21:10:48.265988 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:48.265955    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:51.591150 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:51.591116    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:10:52.826789 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.826754    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"]
Nov 03 21:10:52.830839 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.830818    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.856229 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.856191    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69n5\" (UniqueName: \"kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.856385 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.856253    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.856385 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.856355    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.856487 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.856389    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.879681 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.879649    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-10fa59-serving-cert\""
Nov 03 21:10:52.928517 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.928475    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"]
Nov 03 21:10:52.957578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.957541    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69n5\" (UniqueName: \"kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.957774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.957596    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.957774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.957640    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.957774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.957664    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.957774 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:52.957766    5318 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-serving-cert: secret "maas-api-e2e-api-b-10fa59-serving-cert" not found
Nov 03 21:10:52.957995 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:52.957842    5318 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls podName:acea590a-9071-4341-8096-c1b68b2788c1 nodeName:}" failed. No retries permitted until 2026-11-03 21:10:53.457825692 +0000 UTC m=+2446.437382514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls") pod "maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" (UID: "acea590a-9071-4341-8096-c1b68b2788c1") : secret "maas-api-e2e-api-b-10fa59-serving-cert" not found
Nov 03 21:10:52.958407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.958388    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:52.960279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:52.960252    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:53.015550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:53.015516    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69n5\" (UniqueName: \"kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:53.094450 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:53.094357    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:53.298561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:53.298525    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:53.463447 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:53.463361    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:53.465861 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:53.465834    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\") pod \"maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:53.744041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:53.743953    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:54.115958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:54.115931    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"]
Nov 03 21:10:54.118979 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:10:54.118946    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea590a_9071_4341_8096_c1b68b2788c1.slice/crio-1341eac3562378e53b4b09abdcdd5b6cc8daf7e8e1064aa27dbde60680b1a33f WatchSource:0}: Error finding container 1341eac3562378e53b4b09abdcdd5b6cc8daf7e8e1064aa27dbde60680b1a33f: Status 404 returned error can't find the container with id 1341eac3562378e53b4b09abdcdd5b6cc8daf7e8e1064aa27dbde60680b1a33f
Nov 03 21:10:54.642198 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:54.642164    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" event={"ID":"acea590a-9071-4341-8096-c1b68b2788c1","Type":"ContainerStarted","Data":"1341eac3562378e53b4b09abdcdd5b6cc8daf7e8e1064aa27dbde60680b1a33f"}
Nov 03 21:10:55.649827 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:55.649794    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" event={"ID":"acea590a-9071-4341-8096-c1b68b2788c1","Type":"ContainerStarted","Data":"74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a"}
Nov 03 21:10:55.650237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:55.649919    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:10:55.762757 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:55.762705    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" podStartSLOduration=2.891322696 podStartE2EDuration="3.762691652s" totalImagesPullingTime="871.368956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:55.760433811 +0000 UTC m=+2448.739990651" watchObservedRunningTime="2026-11-03 21:10:55.762691652 +0000 UTC m=+2448.742248492"
Nov 03 21:10:55.934796 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:55.934706    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:10:57.593807 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.593775    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"]
Nov 03 21:10:57.594300 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.593988    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" podUID="dd9837ff-b402-4a39-a327-0017091b0f43" containerName="maas-api" containerID="cri-o://eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86" gracePeriod=30
Nov 03 21:10:57.756868 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.756821    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:57.875980 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.875952    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 21:10:57.920983 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.920947    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs\") pod \"dd9837ff-b402-4a39-a327-0017091b0f43\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") "
Nov 03 21:10:57.921185 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.921061    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\") pod \"dd9837ff-b402-4a39-a327-0017091b0f43\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") "
Nov 03 21:10:57.921185 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.921157    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s\" (UniqueName: \"kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s\") pod \"dd9837ff-b402-4a39-a327-0017091b0f43\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") "
Nov 03 21:10:57.921185 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.921180    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca\") pod \"dd9837ff-b402-4a39-a327-0017091b0f43\" (UID: \"dd9837ff-b402-4a39-a327-0017091b0f43\") "
Nov 03 21:10:57.921635 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.921606    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca" pod "dd9837ff-b402-4a39-a327-0017091b0f43" (UID: "dd9837ff-b402-4a39-a327-0017091b0f43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:10:57.923723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.923697    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs" pod "dd9837ff-b402-4a39-a327-0017091b0f43" (UID: "dd9837ff-b402-4a39-a327-0017091b0f43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:57.923841 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.923721    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls" pod "dd9837ff-b402-4a39-a327-0017091b0f43" (UID: "dd9837ff-b402-4a39-a327-0017091b0f43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:10:57.923841 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:57.923811    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s" pod "dd9837ff-b402-4a39-a327-0017091b0f43" (UID: "dd9837ff-b402-4a39-a327-0017091b0f43"). InnerVolumeSpecName "kube-api-access-sdz7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:10:58.024574 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.024541    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:58.024574 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.024571    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdz7s\" (UniqueName: \"kubernetes.io/projected/dd9837ff-b402-4a39-a327-0017091b0f43-kube-api-access-sdz7s\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:58.024574 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.024581    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9837ff-b402-4a39-a327-0017091b0f43-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:58.024828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.024593    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9837ff-b402-4a39-a327-0017091b0f43-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:10:58.672652 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.672619    5318 generic.go:356] "Generic (PLEG): container finished" podID="dd9837ff-b402-4a39-a327-0017091b0f43" containerID="eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86" exitCode=0
Nov 03 21:10:58.673137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.672684    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"
Nov 03 21:10:58.673137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.672706    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" event={"ID":"dd9837ff-b402-4a39-a327-0017091b0f43","Type":"ContainerDied","Data":"eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86"}
Nov 03 21:10:58.673137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.672793    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz" event={"ID":"dd9837ff-b402-4a39-a327-0017091b0f43","Type":"ContainerDied","Data":"a5085b7d5461a48109dbcf7969eef8a518d3dfb3fe007b566b3183534d5ed61e"}
Nov 03 21:10:58.673137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.672821    5318 scope.go:122] "RemoveContainer" containerID="eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86"
Nov 03 21:10:58.685406 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.685384    5318 scope.go:122] "RemoveContainer" containerID="eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86"
Nov 03 21:10:58.685713 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:10:58.685686    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86\": container with ID starting with eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86 not found: ID does not exist" containerID="eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86"
Nov 03 21:10:58.685822 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.685724    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86"} err="failed to get container status \"eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86\": rpc error: code = NotFound desc = could not find container \"eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86\": container with ID starting with eaee09665692ac6ff6d782ccd70bf9c8a009f6c6611e698e189ba10dd4a3cf86 not found: ID does not exist"
Nov 03 21:10:58.827781 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.827745    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"]
Nov 03 21:10:58.874315 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:58.874276    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2a5ec9-784d6bfd79-9l5bz"]
Nov 03 21:10:59.257599 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:59.257567    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9837ff-b402-4a39-a327-0017091b0f43" path="/var/lib/kubelet/pods/dd9837ff-b402-4a39-a327-0017091b0f43/volumes"
Nov 03 21:10:59.481926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:59.481891    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:10:59.718769 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:10:59.718741    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.263562 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.263526    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.633835 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.633797    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"]
Nov 03 21:11:00.634329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.634314    5318 cpu_manager.go:414] "containerMap: removing container" podUID="dd9837ff-b402-4a39-a327-0017091b0f43" containerName="maas-api"
Nov 03 21:11:00.634400 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.634331    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9837ff-b402-4a39-a327-0017091b0f43" containerName="maas-api"
Nov 03 21:11:00.634500 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.634490    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9837ff-b402-4a39-a327-0017091b0f43" containerName="maas-api"
Nov 03 21:11:00.637772 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.637749    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.690556 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.690525    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df71d7\"/\"kube-root-ca.crt\""
Nov 03 21:11:00.735239 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.735206    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df71d7\"/\"openshift-service-ca.crt\""
Nov 03 21:11:00.735768 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.735583    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df71d7\"/\"default-dockercfg-6sgqn\""
Nov 03 21:11:00.735768 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.735642    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df71d7\"/\"test-model-df71d7-kserve-self-signed-certs\""
Nov 03 21:11:00.739171 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.739151    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"]
Nov 03 21:11:00.755599 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.755571    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.755712 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.755605    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.755786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.755765    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.755930 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.755915    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.755998 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.755981    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2fd\" (UniqueName: \"kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.756064 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.756050    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.808831 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.808775    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:00.857617 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.857585    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.857795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.857651    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2fd\" (UniqueName: \"kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.857795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.857682    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.857795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.857701    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.857795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.857722    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.858012 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.857815    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.858073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.858046    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.858073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.858057    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.858212 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.858185    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.860709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.860686    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.860838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.860800    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.915150 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.915023    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2fd\" (UniqueName: \"kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd\") pod \"test-model-df71d7-kserve-b7ff8fd69-tz99g\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") " pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:00.950105 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:00.950051    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:01.338912 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:01.338885    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"]
Nov 03 21:11:01.340739 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:11:01.340712    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde86dad5_1f13_46a5_87f0_21cb0452229e.slice/crio-04bd511c652904ba5a8ab37a2421fcc992d34febe43a8c7e6f4d56cafb5d9003 WatchSource:0}: Error finding container 04bd511c652904ba5a8ab37a2421fcc992d34febe43a8c7e6f4d56cafb5d9003: Status 404 returned error can't find the container with id 04bd511c652904ba5a8ab37a2421fcc992d34febe43a8c7e6f4d56cafb5d9003
Nov 03 21:11:01.660480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:01.660411    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:11:01.697854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:01.697821    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g" event={"ID":"de86dad5-1f13-46a5-87f0-21cb0452229e","Type":"ContainerStarted","Data":"02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a"}
Nov 03 21:11:01.698004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:01.697865    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g" event={"ID":"de86dad5-1f13-46a5-87f0-21cb0452229e","Type":"ContainerStarted","Data":"04bd511c652904ba5a8ab37a2421fcc992d34febe43a8c7e6f4d56cafb5d9003"}
Nov 03 21:11:01.919003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:01.918897    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g" podStartSLOduration=1.918880916 podStartE2EDuration="1.918880916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:01.918037416 +0000 UTC m=+2454.897594256" watchObservedRunningTime="2026-11-03 21:11:01.918880916 +0000 UTC m=+2454.898437757"
Nov 03 21:11:01.950466 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:01.950437    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:02.577962 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:02.577928    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:02.826289 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:02.826256    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:05.149761 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:05.149725    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:06.550297 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:06.550264    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:06.756256 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:06.756206    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:06.963496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:06.963467    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:07.730841 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:07.730802    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:07.986054 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:07.985988    5318 scope.go:122] "RemoveContainer" containerID="7769d32d509b15be0ed1f3a0fc371c33904ac5a972e302a446ade68e5dfc5551"
Nov 03 21:11:07.995103 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:07.995079    5318 scope.go:122] "RemoveContainer" containerID="ccae702400906690027e65f1d6e8f6c587888f0357337425f06d250d5f829bdc"
Nov 03 21:11:09.688542 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:09.688513    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:11:10.500963 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:10.500903    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:10.950245 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:10.950211    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:10.962924 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:10.962899    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:11.781529 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:11.781501    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:11:12.659758 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:12.659726    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"]
Nov 03 21:11:12.705620 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:12.705598    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"]
Nov 03 21:11:12.866973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:12.866942    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:15.756427 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:15.756391    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:20.144564 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:20.144532    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:20.353283 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:20.353249    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:20.564403 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:20.564313    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:20.772061 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:20.772022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:22.053969 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:22.053935    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:23.539417 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.539381    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"]
Nov 03 21:11:23.544771 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.544748    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.632894 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.632853    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2n66\" (UniqueName: \"kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.633081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.632907    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.633081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.632984    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.633184 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.633150    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.633225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.633191    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.633265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.633253    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.639560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.639529    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fc4e8d\"/\"default-dockercfg-nfdw9\""
Nov 03 21:11:23.639560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.639543    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fc4e8d\"/\"openshift-service-ca.crt\""
Nov 03 21:11:23.639731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.639588    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fc4e8d\"/\"test-model-fc4e8d-kserve-self-signed-certs\""
Nov 03 21:11:23.639731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.639613    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fc4e8d\"/\"kube-root-ca.crt\""
Nov 03 21:11:23.642277 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.642252    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"]
Nov 03 21:11:23.696202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.696147    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:23.734213 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734183    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2n66\" (UniqueName: \"kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734219    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734285    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734367    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734412    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734453    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734724 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734697    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734812 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734790    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.734863 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.734835    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.736530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.736511    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.736934 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.736915    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.791907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.791831    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2n66\" (UniqueName: \"kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66\") pod \"test-model-fc4e8d-kserve-f47d6fdf5-wtt6q\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") " pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:23.857534 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:23.857494    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:24.233558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:24.233532    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"]
Nov 03 21:11:24.235521 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:11:24.235475    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b8db9e4_0ca4_4d87_8789_271ea2601896.slice/crio-84d4a9d400b18eb0a411150d7a56da3a15264d7f625a109b02f56d9f3e98f378 WatchSource:0}: Error finding container 84d4a9d400b18eb0a411150d7a56da3a15264d7f625a109b02f56d9f3e98f378: Status 404 returned error can't find the container with id 84d4a9d400b18eb0a411150d7a56da3a15264d7f625a109b02f56d9f3e98f378
Nov 03 21:11:24.856113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:24.856076    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q" event={"ID":"1b8db9e4-0ca4-4d87-8789-271ea2601896","Type":"ContainerStarted","Data":"26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910"}
Nov 03 21:11:24.856113 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:24.856121    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q" event={"ID":"1b8db9e4-0ca4-4d87-8789-271ea2601896","Type":"ContainerStarted","Data":"84d4a9d400b18eb0a411150d7a56da3a15264d7f625a109b02f56d9f3e98f378"}
Nov 03 21:11:24.858361 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:24.858328    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:24.965584 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:24.965537    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q" podStartSLOduration=1.965522863 podStartE2EDuration="1.965522863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:24.965507703 +0000 UTC m=+2477.945064540" watchObservedRunningTime="2026-11-03 21:11:24.965522863 +0000 UTC m=+2477.945079702"
Nov 03 21:11:25.855812 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:25.855757    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:25.934573 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:11:25.934535    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:11:27.213759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.213718    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:27.680772 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.680734    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main" containerID="cri-o://d254e2d02671152ae92a00c192c70cc3ef87e31e438ae17390c0313b72c57c63" gracePeriod=45
Nov 03 21:11:27.724998 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.724957    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main" containerID="cri-o://dcb4c7e6326d46b461e3f494ca85aa4cd45d742136da970a8bb2416379e9640a" gracePeriod=45
Nov 03 21:11:27.878557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.878367    5318 generic.go:356] "Generic (PLEG): container finished" podID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerID="d254e2d02671152ae92a00c192c70cc3ef87e31e438ae17390c0313b72c57c63" exitCode=0
Nov 03 21:11:27.878557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.878417    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" event={"ID":"55cc4d93-0a99-4379-b682-33fc3fed8524","Type":"ContainerDied","Data":"d254e2d02671152ae92a00c192c70cc3ef87e31e438ae17390c0313b72c57c63"}
Nov 03 21:11:27.881003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.880881    5318 generic.go:356] "Generic (PLEG): container finished" podID="468fa626-3d72-4253-b637-4a6c06e954f4" containerID="dcb4c7e6326d46b461e3f494ca85aa4cd45d742136da970a8bb2416379e9640a" exitCode=0
Nov 03 21:11:27.881003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:27.880958    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" event={"ID":"468fa626-3d72-4253-b637-4a6c06e954f4","Type":"ContainerDied","Data":"dcb4c7e6326d46b461e3f494ca85aa4cd45d742136da970a8bb2416379e9640a"}
Nov 03 21:11:28.015852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.015530    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 21:11:28.058845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.058816    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086628    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache\") pod \"55cc4d93-0a99-4379-b682-33fc3fed8524\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086690    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs\") pod \"55cc4d93-0a99-4379-b682-33fc3fed8524\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086726    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home\") pod \"468fa626-3d72-4253-b637-4a6c06e954f4\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086753    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home\") pod \"55cc4d93-0a99-4379-b682-33fc3fed8524\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086805    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir\") pod \"55cc4d93-0a99-4379-b682-33fc3fed8524\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086901    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8\" (UniqueName: \"kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8\") pod \"55cc4d93-0a99-4379-b682-33fc3fed8524\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.086962    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir\") pod \"468fa626-3d72-4253-b637-4a6c06e954f4\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.087009    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m\" (UniqueName: \"kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m\") pod \"468fa626-3d72-4253-b637-4a6c06e954f4\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.087045    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs\") pod \"468fa626-3d72-4253-b637-4a6c06e954f4\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.087093    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm\") pod \"468fa626-3d72-4253-b637-4a6c06e954f4\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.087156    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache\") pod \"468fa626-3d72-4253-b637-4a6c06e954f4\" (UID: \"468fa626-3d72-4253-b637-4a6c06e954f4\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.087291    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm\") pod \"55cc4d93-0a99-4379-b682-33fc3fed8524\" (UID: \"55cc4d93-0a99-4379-b682-33fc3fed8524\") "
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.090894    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm" pod "55cc4d93-0a99-4379-b682-33fc3fed8524" (UID: "55cc4d93-0a99-4379-b682-33fc3fed8524"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.091260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.091107    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache" pod "55cc4d93-0a99-4379-b682-33fc3fed8524" (UID: "55cc4d93-0a99-4379-b682-33fc3fed8524"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.093485 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.093449    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home" pod "468fa626-3d72-4253-b637-4a6c06e954f4" (UID: "468fa626-3d72-4253-b637-4a6c06e954f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.093692 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.093670    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs" pod "55cc4d93-0a99-4379-b682-33fc3fed8524" (UID: "55cc4d93-0a99-4379-b682-33fc3fed8524"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:11:28.094049 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.094021    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home" pod "55cc4d93-0a99-4379-b682-33fc3fed8524" (UID: "55cc4d93-0a99-4379-b682-33fc3fed8524"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.094318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.094287    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir" pod "55cc4d93-0a99-4379-b682-33fc3fed8524" (UID: "55cc4d93-0a99-4379-b682-33fc3fed8524"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.094527 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.094509    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir" pod "468fa626-3d72-4253-b637-4a6c06e954f4" (UID: "468fa626-3d72-4253-b637-4a6c06e954f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.095998 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.095106    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache" pod "468fa626-3d72-4253-b637-4a6c06e954f4" (UID: "468fa626-3d72-4253-b637-4a6c06e954f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.096162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.096136    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m" pod "468fa626-3d72-4253-b637-4a6c06e954f4" (UID: "468fa626-3d72-4253-b637-4a6c06e954f4"). InnerVolumeSpecName "kube-api-access-wrp8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:28.096814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.096792    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm" pod "468fa626-3d72-4253-b637-4a6c06e954f4" (UID: "468fa626-3d72-4253-b637-4a6c06e954f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:11:28.097272 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.097206    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs" pod "468fa626-3d72-4253-b637-4a6c06e954f4" (UID: "468fa626-3d72-4253-b637-4a6c06e954f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:11:28.097946 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.097918    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8" pod "55cc4d93-0a99-4379-b682-33fc3fed8524" (UID: "55cc4d93-0a99-4379-b682-33fc3fed8524"). InnerVolumeSpecName "kube-api-access-nq5c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189021    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189051    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189061    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc4d93-0a99-4379-b682-33fc3fed8524-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189070    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189078    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189086    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc4d93-0a99-4379-b682-33fc3fed8524-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189095    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq5c8\" (UniqueName: \"kubernetes.io/projected/55cc4d93-0a99-4379-b682-33fc3fed8524-kube-api-access-nq5c8\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189104    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189112    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrp8m\" (UniqueName: \"kubernetes.io/projected/468fa626-3d72-4253-b637-4a6c06e954f4-kube-api-access-wrp8m\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189120    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/468fa626-3d72-4253-b637-4a6c06e954f4-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189128    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.189484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.189135    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/468fa626-3d72-4253-b637-4a6c06e954f4-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:28.889417 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.889387    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"
Nov 03 21:11:28.889878 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.889387    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42" event={"ID":"55cc4d93-0a99-4379-b682-33fc3fed8524","Type":"ContainerDied","Data":"6da0e4ebe04ecec2141ee3a258d7723b5df46a55d7f5c98df5a132626fc1f0db"}
Nov 03 21:11:28.889878 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.889525    5318 scope.go:122] "RemoveContainer" containerID="d254e2d02671152ae92a00c192c70cc3ef87e31e438ae17390c0313b72c57c63"
Nov 03 21:11:28.891046 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.891024    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb" event={"ID":"468fa626-3d72-4253-b637-4a6c06e954f4","Type":"ContainerDied","Data":"13d5625d896c916d9d2f43c650524220850ecc7a98ac4e3fefe72af46b318872"}
Nov 03 21:11:28.891154 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.891056    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"
Nov 03 21:11:28.900911 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:28.900894    5318 scope.go:122] "RemoveContainer" containerID="dcb4c7e6326d46b461e3f494ca85aa4cd45d742136da970a8bb2416379e9640a"
Nov 03 21:11:29.043896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:29.043815    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"]
Nov 03 21:11:29.089761 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:29.089726    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/simulator-w0-1e6fe1-kserve-69cc484575-kskmb"]
Nov 03 21:11:29.150796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:29.150765    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"]
Nov 03 21:11:29.156980 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:29.156957    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1e6fe1/distinct-w0-1e6fe1-kserve-69554b6fc7-2bf42"]
Nov 03 21:11:29.256513 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:29.256484    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" path="/var/lib/kubelet/pods/468fa626-3d72-4253-b637-4a6c06e954f4/volumes"
Nov 03 21:11:29.256980 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:29.256965    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" path="/var/lib/kubelet/pods/55cc4d93-0a99-4379-b682-33fc3fed8524/volumes"
Nov 03 21:11:30.817190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:30.817152    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:33.858588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:33.858555    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:33.871391 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:33.871366    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:33.935798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:33.935775    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:11:35.814559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:35.814524    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:39.328008 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:39.327976    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:11:41.199264 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:41.199227    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:46.542218 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:46.542181    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:50.101858 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:50.101791    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:50.313849 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:50.313816    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:50.961060 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:50.961026    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:51.737729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:51.737691    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:54.871390 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:54.871329    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:55.412294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:55.412260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:55.968264 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:11:55.968208    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:11:56.050728 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:56.050693    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:56.531783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:56.531756    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:57.329373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:57.329306    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:57.615907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:57.615869    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:11:59.210598 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.210568    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"]
Nov 03 21:11:59.211088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.211075    5318 cpu_manager.go:414] "containerMap: removing container" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main"
Nov 03 21:11:59.211131 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.211089    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main"
Nov 03 21:11:59.211131 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.211106    5318 cpu_manager.go:414] "containerMap: removing container" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main"
Nov 03 21:11:59.211131 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.211111    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main"
Nov 03 21:11:59.211225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.211209    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="55cc4d93-0a99-4379-b682-33fc3fed8524" containerName="main"
Nov 03 21:11:59.211225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.211220    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="468fa626-3d72-4253-b637-4a6c06e954f4" containerName="main"
Nov 03 21:11:59.215684 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.215659    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.304816 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.304780    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"]
Nov 03 21:11:59.346582 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.346546    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f1855-openshift-default-dockercfg-n94fj\""
Nov 03 21:11:59.348280 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348240    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348455 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348296    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348450    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348489    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348600    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mtb\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348662 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348640    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348705    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348923 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348902    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.348988 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.348973    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.449990    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450026    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450064    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mtb\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450087    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450110    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450182    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450226    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450515 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450466    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450553    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450571    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450591    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450740 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450639    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450816    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.450957 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.450932    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.452742 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.452719    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.452820 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.452789    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.508120 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.508059    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mtb\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.508205 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.508146    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.529588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.529560    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:11:59.948123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.948094    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"]
Nov 03 21:11:59.950043 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:11:59.950015    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3c10cb_8e12_4f51_80f9_ab41f5cb140b.slice/crio-1b90e54754fd9dfa6587216e1c8f372ec31f010fbb9d4a6dadafe874113fa7c5 WatchSource:0}: Error finding container 1b90e54754fd9dfa6587216e1c8f372ec31f010fbb9d4a6dadafe874113fa7c5: Status 404 returned error can't find the container with id 1b90e54754fd9dfa6587216e1c8f372ec31f010fbb9d4a6dadafe874113fa7c5
Nov 03 21:11:59.952789 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.952757    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:59.952880 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.952822    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:11:59.952880 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:11:59.952849    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:12:00.132298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:00.132248    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" event={"ID":"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b","Type":"ContainerStarted","Data":"5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9"}
Nov 03 21:12:00.132515 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:00.132316    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" event={"ID":"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b","Type":"ContainerStarted","Data":"1b90e54754fd9dfa6587216e1c8f372ec31f010fbb9d4a6dadafe874113fa7c5"}
Nov 03 21:12:00.247962 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:00.247864    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podStartSLOduration=1.247847852 podStartE2EDuration="1.247847852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:00.246547694 +0000 UTC m=+2513.226104527" watchObservedRunningTime="2026-11-03 21:12:00.247847852 +0000 UTC m=+2513.227404692"
Nov 03 21:12:00.530362 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:00.530277    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:12:00.746174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:00.746144    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:00.961105 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:00.961072    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:01.192099 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:01.191866    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:01.532047 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:01.532010    5318 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 03 21:12:01.532460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:01.532066    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 03 21:12:02.529921 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:02.529884    5318 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 03 21:12:02.530104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:02.529939    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 03 21:12:03.315415 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:03.315380    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:03.530348 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:03.530298    5318 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 03 21:12:03.530522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:03.530379    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 03 21:12:04.530295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:04.530254    5318 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 03 21:12:04.530789 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:04.530415    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 03 21:12:04.992971 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:04.992940    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:05.530762 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:05.530727    5318 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 03 21:12:05.531357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:05.531305    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 03 21:12:06.536389 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:06.536353    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:12:06.536868 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:06.536744    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:12:06.537445 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:06.537424    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:12:06.706185 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:06.706127    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:08.300116 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:08.300038    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:08.545616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:08.545576    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:09.165676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:09.165623    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:10.850993 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:10.850887    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:11.067578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:11.067547    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:14.437051 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:14.437022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:19.505405 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:19.505296    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:24.486646 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:24.486613    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:25.933886 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:12:25.933850    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:12:27.656009 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:27.655975    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:28.268860 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:28.268830    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:12:29.815101 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:29.815060    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:34.437240 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:34.437197    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:35.471364 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:35.471296    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:40.450890 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:40.450858    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:40.688886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:40.688853    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:41.007006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:41.006969    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:45.632300 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:45.632264    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"]
Nov 03 21:12:45.632698 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:45.632521    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" podUID="409352e1-86cb-4b04-b253-6f6e8301576a" containerName="maas-api" containerID="cri-o://ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388" gracePeriod=30
Nov 03 21:12:45.799068 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:45.798969    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:45.929555 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:45.929531    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 21:12:46.008008 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.007980    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca\") pod \"409352e1-86cb-4b04-b253-6f6e8301576a\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") "
Nov 03 21:12:46.008199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.008070    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs\") pod \"409352e1-86cb-4b04-b253-6f6e8301576a\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") "
Nov 03 21:12:46.008199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.008097    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\") pod \"409352e1-86cb-4b04-b253-6f6e8301576a\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") "
Nov 03 21:12:46.008890 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.008312    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr\") pod \"409352e1-86cb-4b04-b253-6f6e8301576a\" (UID: \"409352e1-86cb-4b04-b253-6f6e8301576a\") "
Nov 03 21:12:46.008890 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.008760    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca" pod "409352e1-86cb-4b04-b253-6f6e8301576a" (UID: "409352e1-86cb-4b04-b253-6f6e8301576a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:46.013991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.009257    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/409352e1-86cb-4b04-b253-6f6e8301576a-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:46.013991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.011089    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr" pod "409352e1-86cb-4b04-b253-6f6e8301576a" (UID: "409352e1-86cb-4b04-b253-6f6e8301576a"). InnerVolumeSpecName "kube-api-access-kwwvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:46.020455 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.020425    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs" pod "409352e1-86cb-4b04-b253-6f6e8301576a" (UID: "409352e1-86cb-4b04-b253-6f6e8301576a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:46.021664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.021642    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls" pod "409352e1-86cb-4b04-b253-6f6e8301576a" (UID: "409352e1-86cb-4b04-b253-6f6e8301576a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:46.110524 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.110491    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:46.110524 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.110519    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/409352e1-86cb-4b04-b253-6f6e8301576a-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:46.110720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.110533    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwwvr\" (UniqueName: \"kubernetes.io/projected/409352e1-86cb-4b04-b253-6f6e8301576a-kube-api-access-kwwvr\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:46.457236 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.457203    5318 generic.go:356] "Generic (PLEG): container finished" podID="409352e1-86cb-4b04-b253-6f6e8301576a" containerID="ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388" exitCode=0
Nov 03 21:12:46.457454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.457251    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"
Nov 03 21:12:46.457454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.457283    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" event={"ID":"409352e1-86cb-4b04-b253-6f6e8301576a","Type":"ContainerDied","Data":"ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388"}
Nov 03 21:12:46.457454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.457352    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx" event={"ID":"409352e1-86cb-4b04-b253-6f6e8301576a","Type":"ContainerDied","Data":"620929c1de3b9589b0b082c38e50bc3fead4fcc505cf7820cf3253e0231cbe89"}
Nov 03 21:12:46.457454 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.457354    5318 scope.go:122] "RemoveContainer" containerID="ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388"
Nov 03 21:12:46.469013 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.468997    5318 scope.go:122] "RemoveContainer" containerID="ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388"
Nov 03 21:12:46.469282 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:12:46.469260    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388\": container with ID starting with ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388 not found: ID does not exist" containerID="ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388"
Nov 03 21:12:46.469398 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.469286    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388"} err="failed to get container status \"ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388\": rpc error: code = NotFound desc = could not find container \"ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388\": container with ID starting with ccd29c80137f56d106d6caddeeac6fb9157248dd8612236821b14c4365764388 not found: ID does not exist"
Nov 03 21:12:46.612774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.612742    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"]
Nov 03 21:12:46.618922 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:46.618891    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1e6fe1-688fb49b85-kldlx"]
Nov 03 21:12:47.259789 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:47.259757    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409352e1-86cb-4b04-b253-6f6e8301576a" path="/var/lib/kubelet/pods/409352e1-86cb-4b04-b253-6f6e8301576a/volumes"
Nov 03 21:12:47.465422 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:47.465393    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:47.717186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:47.717149    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:47.929071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:47.929044    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:48.436274 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:48.436246    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:53.214217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:53.214185    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:54.668294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:54.668261    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:54.898785 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:54.898739    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:55.118285 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:55.118216    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:12:55.934855 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:12:55.934819    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:12:58.232762 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:12:58.232731    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:00.192891 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:00.192851    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:13:03.104948 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:03.104915    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:03.312607 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:03.312571    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:08.060375 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:08.060327    5318 scope.go:122] "RemoveContainer" containerID="8549554a7867a4ebc523faeac30bb7f4e1c216b3fa47d8d23485d60169a6311f"
Nov 03 21:13:08.069548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:08.069521    5318 scope.go:122] "RemoveContainer" containerID="d3a551ed1ccd8021a66131dabd091a7272e3f9e41ca80413e37f6bb7979c94a7"
Nov 03 21:13:08.308029 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:08.308002    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:13.262564 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:13.262533    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:18.207196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:18.207112    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:23.271048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:23.271012    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:25.936379 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:13:25.936325    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:13:27.929702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:27.929672    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:28.136036 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:28.136008    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:33.214438 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:33.214405    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:37.549532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:37.549495    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:39.356119 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:39.356082    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:39.575688 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:39.575654    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:39.835429 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:39.835355    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:40.567384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:40.567310    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:44.053852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:44.053813    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"]
Nov 03 21:13:44.100683 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:44.100647    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8532db3-7rbxh"]
Nov 03 21:13:44.700540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:44.700505    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:45.255964 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:45.255932    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5773541-4239-4aa9-b981-3939bcc5dbcc" path="/var/lib/kubelet/pods/a5773541-4239-4aa9-b981-3939bcc5dbcc/volumes"
Nov 03 21:13:46.615409 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:46.615363    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:46.685783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:46.685753    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:13:46.840946 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:46.840912    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:47.185134 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:47.185101    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:49.942186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:49.942152    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:55.246438 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:55.246403    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:13:55.934754 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:13:55.934722    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:13:59.797858 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:13:59.797819    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:00.003135 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:00.003107    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:00.433110 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:00.433082    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:05.423404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:05.423371    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:08.349558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:08.349522    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:09.874031 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:09.873997    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"]
Nov 03 21:14:09.991108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:09.991068    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl" podUID="a174b6b5-a6d0-4f7e-a362-552b9c76a47e" containerName="maas-api" containerID="cri-o://4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34" gracePeriod=30
Nov 03 21:14:10.080870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.080800    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:10.298568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.298529    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:14:10.376709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.376679    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\") pod \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") "
Nov 03 21:14:10.376911 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.376763    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv\" (UniqueName: \"kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv\") pod \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") "
Nov 03 21:14:10.376982 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.376947    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs\") pod \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") "
Nov 03 21:14:10.377059 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.377042    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca\") pod \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\" (UID: \"a174b6b5-a6d0-4f7e-a362-552b9c76a47e\") "
Nov 03 21:14:10.377419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.377391    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca" pod "a174b6b5-a6d0-4f7e-a362-552b9c76a47e" (UID: "a174b6b5-a6d0-4f7e-a362-552b9c76a47e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:10.379127 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.379101    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv" pod "a174b6b5-a6d0-4f7e-a362-552b9c76a47e" (UID: "a174b6b5-a6d0-4f7e-a362-552b9c76a47e"). InnerVolumeSpecName "kube-api-access-h67gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:10.379231 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.379184    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs" pod "a174b6b5-a6d0-4f7e-a362-552b9c76a47e" (UID: "a174b6b5-a6d0-4f7e-a362-552b9c76a47e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:14:10.379298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.379264    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls" pod "a174b6b5-a6d0-4f7e-a362-552b9c76a47e" (UID: "a174b6b5-a6d0-4f7e-a362-552b9c76a47e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:14:10.478798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.478716    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:10.478798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.478748    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:10.478798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.478762    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:10.478798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.478774    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h67gv\" (UniqueName: \"kubernetes.io/projected/a174b6b5-a6d0-4f7e-a362-552b9c76a47e-kube-api-access-h67gv\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:10.998991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.998959    5318 generic.go:356] "Generic (PLEG): container finished" podID="a174b6b5-a6d0-4f7e-a362-552b9c76a47e" containerID="4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34" exitCode=0
Nov 03 21:14:10.999419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.999019    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"
Nov 03 21:14:10.999419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.999056    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl" event={"ID":"a174b6b5-a6d0-4f7e-a362-552b9c76a47e","Type":"ContainerDied","Data":"4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34"}
Nov 03 21:14:10.999419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.999154    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl" event={"ID":"a174b6b5-a6d0-4f7e-a362-552b9c76a47e","Type":"ContainerDied","Data":"91c119977860bbd62b4567d204a9ca3c77e579e9992556c234b7b5dcc822de63"}
Nov 03 21:14:10.999419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:10.999187    5318 scope.go:122] "RemoveContainer" containerID="4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34"
Nov 03 21:14:11.012064 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.011753    5318 scope.go:122] "RemoveContainer" containerID="4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34"
Nov 03 21:14:11.012434 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:14:11.012325    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34\": container with ID starting with 4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34 not found: ID does not exist" containerID="4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34"
Nov 03 21:14:11.012434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.012378    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34"} err="failed to get container status \"4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34\": rpc error: code = NotFound desc = could not find container \"4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34\": container with ID starting with 4ae479d7d65f8e3279d9d42d636c1aecf519110ff6e304d800816b89e99efe34 not found: ID does not exist"
Nov 03 21:14:11.152992 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.152968    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"]
Nov 03 21:14:11.198841 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.198817    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8afb98-766c65c99-qljwl"]
Nov 03 21:14:11.255438 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.255358    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a174b6b5-a6d0-4f7e-a362-552b9c76a47e" path="/var/lib/kubelet/pods/a174b6b5-a6d0-4f7e-a362-552b9c76a47e/volumes"
Nov 03 21:14:11.680760 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.680727    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:11.911558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:11.911528    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:12.117519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:12.117488    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:12.633066 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:12.633036    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:17.072517 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:17.072483    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:17.293384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:17.293352    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:18.638993 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:18.638959    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:18.883768 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:18.883718    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:19.104162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:19.104128    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:21.273081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273050    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"]
Nov 03 21:14:21.273786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273765    5318 cpu_manager.go:414] "containerMap: removing container" podUID="409352e1-86cb-4b04-b253-6f6e8301576a" containerName="maas-api"
Nov 03 21:14:21.273883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273787    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409352e1-86cb-4b04-b253-6f6e8301576a" containerName="maas-api"
Nov 03 21:14:21.273883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273808    5318 cpu_manager.go:414] "containerMap: removing container" podUID="a174b6b5-a6d0-4f7e-a362-552b9c76a47e" containerName="maas-api"
Nov 03 21:14:21.273883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273815    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a174b6b5-a6d0-4f7e-a362-552b9c76a47e" containerName="maas-api"
Nov 03 21:14:21.274037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273968    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="409352e1-86cb-4b04-b253-6f6e8301576a" containerName="maas-api"
Nov 03 21:14:21.274037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.273981    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="a174b6b5-a6d0-4f7e-a362-552b9c76a47e" containerName="maas-api"
Nov 03 21:14:21.279977 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.279955    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.331263 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.331231    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f1855\"/\"openshift-service-ca.crt\""
Nov 03 21:14:21.349766 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.349742    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f1855\"/\"kube-root-ca.crt\""
Nov 03 21:14:21.377230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.377202    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f1855\"/\"default-dockercfg-rfvzb\""
Nov 03 21:14:21.377230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.377221    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f1855\"/\"test-model-1f1855-kserve-self-signed-certs\""
Nov 03 21:14:21.379693 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.379673    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"]
Nov 03 21:14:21.396305 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.396277    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhfv\" (UniqueName: \"kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.396460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.396438    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.396522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.396484    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.396606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.396584    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.396746 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.396724    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.397392 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.397372    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.448084 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.448048    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:21.498859 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.498829    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.498999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.498883    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhfv\" (UniqueName: \"kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.498999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.498913    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.498999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.498932    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.498999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.498953    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.499153 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.499109    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.499286 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.499264    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.499387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.499298    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.499501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.499480    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.501606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.501582    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.501807 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.501788    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.556281 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.556215    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhfv\" (UniqueName: \"kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv\") pod \"test-model-1f1855-kserve-99cd448d7-sfv26\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") " pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:21.591530 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:21.591508    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:22.031529 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:22.031503    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"]
Nov 03 21:14:22.034313 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:14:22.033955    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf9e1a77_0705_4e93_80c7_9d7bc980d7b4.slice/crio-300eaed79ad4cf4b747844175c767658d94060afea57b33228b19252a8be7f05 WatchSource:0}: Error finding container 300eaed79ad4cf4b747844175c767658d94060afea57b33228b19252a8be7f05: Status 404 returned error can't find the container with id 300eaed79ad4cf4b747844175c767658d94060afea57b33228b19252a8be7f05
Nov 03 21:14:22.067663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:22.067637    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26" event={"ID":"af9e1a77-0705-4e93-80c7-9d7bc980d7b4","Type":"ContainerStarted","Data":"300eaed79ad4cf4b747844175c767658d94060afea57b33228b19252a8be7f05"}
Nov 03 21:14:22.634312 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:22.634281    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:23.075994 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:23.075902    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26" event={"ID":"af9e1a77-0705-4e93-80c7-9d7bc980d7b4","Type":"ContainerStarted","Data":"4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a"}
Nov 03 21:14:23.186439 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:23.186390    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26" podStartSLOduration=2.186376667 podStartE2EDuration="2.186376667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:23.185017244 +0000 UTC m=+2656.164574086" watchObservedRunningTime="2026-11-03 21:14:23.186376667 +0000 UTC m=+2656.165933769"
Nov 03 21:14:23.592182 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:23.592143    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:25.933073 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:14:25.933034    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:14:27.706009 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:27.705958    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:28.237163 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:28.237130    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:31.244670 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:31.244639    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"]
Nov 03 21:14:31.352057 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:31.352026    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"]
Nov 03 21:14:31.592558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:31.592529    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:31.605441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:31.605417    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:31.678854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:31.678819    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:31.889911 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:31.889830    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:32.149079 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:32.149012    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:14:32.182712 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:32.182681    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:14:32.614663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:32.614633    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:36.245243 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.245188    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl" podUID="75dff506-ee70-4a03-97f6-b692b1fc9bd9" containerName="payload-processing" containerID="cri-o://9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c" gracePeriod=25
Nov 03 21:14:36.352600 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.352570    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt" podUID="42524b01-f10f-4c2d-8d40-47d31b9cf357" containerName="payload-pre-processing" containerID="cri-o://726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f" gracePeriod=25
Nov 03 21:14:36.512824 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.512768    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:14:36.576720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.576674    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx\" (UniqueName: \"kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx\") pod \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") "
Nov 03 21:14:36.576871 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.576779    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume\") pod \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\" (UID: \"75dff506-ee70-4a03-97f6-b692b1fc9bd9\") "
Nov 03 21:14:36.577450 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.577419    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume" pod "75dff506-ee70-4a03-97f6-b692b1fc9bd9" (UID: "75dff506-ee70-4a03-97f6-b692b1fc9bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:36.579488 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.579461    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx" pod "75dff506-ee70-4a03-97f6-b692b1fc9bd9" (UID: "75dff506-ee70-4a03-97f6-b692b1fc9bd9"). InnerVolumeSpecName "kube-api-access-697vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:36.624247 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.624222    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:14:36.678282 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.678248    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq\") pod \"42524b01-f10f-4c2d-8d40-47d31b9cf357\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") "
Nov 03 21:14:36.678452 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.678308    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume\") pod \"42524b01-f10f-4c2d-8d40-47d31b9cf357\" (UID: \"42524b01-f10f-4c2d-8d40-47d31b9cf357\") "
Nov 03 21:14:36.678741 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.678713    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume" pod "42524b01-f10f-4c2d-8d40-47d31b9cf357" (UID: "42524b01-f10f-4c2d-8d40-47d31b9cf357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:36.678862 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.678744    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697vx\" (UniqueName: \"kubernetes.io/projected/75dff506-ee70-4a03-97f6-b692b1fc9bd9-kube-api-access-697vx\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:36.678862 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.678765    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dff506-ee70-4a03-97f6-b692b1fc9bd9-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:36.680416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.680390    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq" pod "42524b01-f10f-4c2d-8d40-47d31b9cf357" (UID: "42524b01-f10f-4c2d-8d40-47d31b9cf357"). InnerVolumeSpecName "kube-api-access-nm7fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:36.779964 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.779843    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/42524b01-f10f-4c2d-8d40-47d31b9cf357-kube-api-access-nm7fq\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:36.779964 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:36.779877    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42524b01-f10f-4c2d-8d40-47d31b9cf357-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:37.169531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.169496    5318 generic.go:356] "Generic (PLEG): container finished" podID="75dff506-ee70-4a03-97f6-b692b1fc9bd9" containerID="9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c" exitCode=0
Nov 03 21:14:37.169718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.169538    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"
Nov 03 21:14:37.169718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.169584    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl" event={"ID":"75dff506-ee70-4a03-97f6-b692b1fc9bd9","Type":"ContainerDied","Data":"9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c"}
Nov 03 21:14:37.169718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.169665    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl" event={"ID":"75dff506-ee70-4a03-97f6-b692b1fc9bd9","Type":"ContainerDied","Data":"aa5f6f6ccf267cf83253fa471765f21fbf7d3fb7ec7017364759260527f9d2d9"}
Nov 03 21:14:37.169718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.169698    5318 scope.go:122] "RemoveContainer" containerID="9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c"
Nov 03 21:14:37.172576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.172454    5318 generic.go:356] "Generic (PLEG): container finished" podID="42524b01-f10f-4c2d-8d40-47d31b9cf357" containerID="726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f" exitCode=0
Nov 03 21:14:37.172576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.172501    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"
Nov 03 21:14:37.172576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.172546    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt" event={"ID":"42524b01-f10f-4c2d-8d40-47d31b9cf357","Type":"ContainerDied","Data":"726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f"}
Nov 03 21:14:37.172801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.172612    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt" event={"ID":"42524b01-f10f-4c2d-8d40-47d31b9cf357","Type":"ContainerDied","Data":"54dea3b13d7072dac639ef28ee54c9ba2452e8f55ac26dff0e5c38bc114a2704"}
Nov 03 21:14:37.182098 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.182076    5318 scope.go:122] "RemoveContainer" containerID="9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c"
Nov 03 21:14:37.182412 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:14:37.182384    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c\": container with ID starting with 9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c not found: ID does not exist" containerID="9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c"
Nov 03 21:14:37.182492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.182421    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c"} err="failed to get container status \"9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c\": rpc error: code = NotFound desc = could not find container \"9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c\": container with ID starting with 9cf6dcd3fa360f12daf94ea2feaf3c7bc1a40b89f4bc48c437342bc83362e85c not found: ID does not exist"
Nov 03 21:14:37.182492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.182443    5318 scope.go:122] "RemoveContainer" containerID="726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f"
Nov 03 21:14:37.193414 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.193389    5318 scope.go:122] "RemoveContainer" containerID="726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f"
Nov 03 21:14:37.193705 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:14:37.193686    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f\": container with ID starting with 726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f not found: ID does not exist" containerID="726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f"
Nov 03 21:14:37.193794 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.193710    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f"} err="failed to get container status \"726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f\": rpc error: code = NotFound desc = could not find container \"726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f\": container with ID starting with 726516fa8dbbe113c1b76743b0484b87d23ad64d4f45fb49544c3fb13554db2f not found: ID does not exist"
Nov 03 21:14:37.323987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.323956    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"]
Nov 03 21:14:37.330383 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.330351    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cfcd54-8489d55c9-9v9nt"]
Nov 03 21:14:37.430727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.430658    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"]
Nov 03 21:14:37.436020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.435990    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cfcd54-7777f4945f-ffsvl"]
Nov 03 21:14:37.671266 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:37.671235    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:39.255030 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:39.254989    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42524b01-f10f-4c2d-8d40-47d31b9cf357" path="/var/lib/kubelet/pods/42524b01-f10f-4c2d-8d40-47d31b9cf357/volumes"
Nov 03 21:14:39.255600 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:39.255579    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75dff506-ee70-4a03-97f6-b692b1fc9bd9" path="/var/lib/kubelet/pods/75dff506-ee70-4a03-97f6-b692b1fc9bd9/volumes"
Nov 03 21:14:42.623965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:42.623934    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:47.676981 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:47.676948    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:52.604631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:52.604594    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:14:55.934896 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:14:55.934858    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:14:57.074116 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:57.074085    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"]
Nov 03 21:14:57.087082 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:57.087057    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e32281-w5zjt"]
Nov 03 21:14:57.255048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:57.255015    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12884503-f8f9-4720-96f7-94d3345939fb" path="/var/lib/kubelet/pods/12884503-f8f9-4720-96f7-94d3345939fb/volumes"
Nov 03 21:14:57.510695 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:14:57.510655    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:00.185792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.185759    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"]
Nov 03 21:15:00.186352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.186323    5318 cpu_manager.go:414] "containerMap: removing container" podUID="75dff506-ee70-4a03-97f6-b692b1fc9bd9" containerName="payload-processing"
Nov 03 21:15:00.186440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.186352    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75dff506-ee70-4a03-97f6-b692b1fc9bd9" containerName="payload-processing"
Nov 03 21:15:00.186440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.186384    5318 cpu_manager.go:414] "containerMap: removing container" podUID="42524b01-f10f-4c2d-8d40-47d31b9cf357" containerName="payload-pre-processing"
Nov 03 21:15:00.186440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.186390    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42524b01-f10f-4c2d-8d40-47d31b9cf357" containerName="payload-pre-processing"
Nov 03 21:15:00.186596 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.186497    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="75dff506-ee70-4a03-97f6-b692b1fc9bd9" containerName="payload-processing"
Nov 03 21:15:00.186596 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.186510    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="42524b01-f10f-4c2d-8d40-47d31b9cf357" containerName="payload-pre-processing"
Nov 03 21:15:00.191277 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.191256    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"]
Nov 03 21:15:00.191411 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.191390    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.195056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.195039    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.202327 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.202304    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"]
Nov 03 21:15:00.206079 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.206062    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.231760 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.231738    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"]
Nov 03 21:15:00.235432 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.235415    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.279667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.279635    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"]
Nov 03 21:15:00.283397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.283364    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"]
Nov 03 21:15:00.330447 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330415    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.330616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330466    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mdk\" (UniqueName: \"kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk\") pod \"maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.330616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330525    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.330616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330558    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6f9c\" (UniqueName: \"kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c\") pod \"maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.331097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330653    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnfnw\" (UniqueName: \"kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw\") pod \"maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.331097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330717    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzkg\" (UniqueName: \"kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.331097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330770    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.331097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.330795    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.331855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.331781    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"]
Nov 03 21:15:00.337919 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.337880    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:15:00.386933 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.386907    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"]
Nov 03 21:15:00.431838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.431801    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6f9c\" (UniqueName: \"kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c\") pod \"maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.432059 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432034    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnfnw\" (UniqueName: \"kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw\") pod \"maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.432142 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432101    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzkg\" (UniqueName: \"kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.432203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432158    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.432430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432404    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.432537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432519    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.432606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432577    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mdk\" (UniqueName: \"kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk\") pod \"maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.432671 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432648    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.432987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.432967    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.433181 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.433157    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.433235 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.433221    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.433301 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.433284    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.489314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.489248    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6f9c\" (UniqueName: \"kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c\") pod \"maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.489314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.489251    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzkg\" (UniqueName: \"kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.489482 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.489385    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnfnw\" (UniqueName: \"kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw\") pod \"maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.489482 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.489423    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mdk\" (UniqueName: \"kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk\") pod \"maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.509940 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.509916    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:00.521825 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.521808    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:00.528977 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.528957    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:00.549858 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.549832    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:00.920197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.920169    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"]
Nov 03 21:15:00.922263 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:15:00.922232    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce542e76_0469_4ffb_877e_b79f024fbfec.slice/crio-c476b48f4edd6f5855b0f3bd0edd40e94a5d217190ffcee5d6e258bd5832a271 WatchSource:0}: Error finding container c476b48f4edd6f5855b0f3bd0edd40e94a5d217190ffcee5d6e258bd5832a271: Status 404 returned error can't find the container with id c476b48f4edd6f5855b0f3bd0edd40e94a5d217190ffcee5d6e258bd5832a271
Nov 03 21:15:00.939416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.939392    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"]
Nov 03 21:15:00.940993 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:15:00.940967    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab85640_f691_46b8_8a5d_53143f4afb69.slice/crio-b27125d963ff20742c3c8b02b6c881b678b92a558a6c22c236b60e5c0bf697a3 WatchSource:0}: Error finding container b27125d963ff20742c3c8b02b6c881b678b92a558a6c22c236b60e5c0bf697a3: Status 404 returned error can't find the container with id b27125d963ff20742c3c8b02b6c881b678b92a558a6c22c236b60e5c0bf697a3
Nov 03 21:15:00.967896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.967872    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"]
Nov 03 21:15:00.978532 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:15:00.978510    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a81ca46_ec2a_4540_823a_897631b027ae.slice/crio-60611cf2b4d46fa8fdc9cdb9070439e00a7eb136c7141d75a4b6a474a595e892 WatchSource:0}: Error finding container 60611cf2b4d46fa8fdc9cdb9070439e00a7eb136c7141d75a4b6a474a595e892: Status 404 returned error can't find the container with id 60611cf2b4d46fa8fdc9cdb9070439e00a7eb136c7141d75a4b6a474a595e892
Nov 03 21:15:00.986733 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:00.986708    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"]
Nov 03 21:15:00.989152 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:15:00.989126    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9437a57f_d923_4cee_bb48_94a536f2964f.slice/crio-0c35816e6fb3dc335b4bb568eb813b393b458266fae21b09017eeb347f2caa68 WatchSource:0}: Error finding container 0c35816e6fb3dc335b4bb568eb813b393b458266fae21b09017eeb347f2caa68: Status 404 returned error can't find the container with id 0c35816e6fb3dc335b4bb568eb813b393b458266fae21b09017eeb347f2caa68
Nov 03 21:15:01.336889 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:01.336853    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" event={"ID":"0ab85640-f691-46b8-8a5d-53143f4afb69","Type":"ContainerStarted","Data":"b27125d963ff20742c3c8b02b6c881b678b92a558a6c22c236b60e5c0bf697a3"}
Nov 03 21:15:01.338290 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:01.338259    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" event={"ID":"0a81ca46-ec2a-4540-823a-897631b027ae","Type":"ContainerStarted","Data":"60611cf2b4d46fa8fdc9cdb9070439e00a7eb136c7141d75a4b6a474a595e892"}
Nov 03 21:15:01.339750 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:01.339723    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" event={"ID":"ce542e76-0469-4ffb-877e-b79f024fbfec","Type":"ContainerStarted","Data":"c476b48f4edd6f5855b0f3bd0edd40e94a5d217190ffcee5d6e258bd5832a271"}
Nov 03 21:15:01.341323 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:01.341301    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerStarted","Data":"7e18ffe66ed396f142fcd4a0ab5efbd416238470573c17be3ae9043c9eeff725"}
Nov 03 21:15:01.341459 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:01.341354    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerStarted","Data":"0c35816e6fb3dc335b4bb568eb813b393b458266fae21b09017eeb347f2caa68"}
Nov 03 21:15:01.451364 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:01.451198    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" podStartSLOduration=1.451183143 podStartE2EDuration="1.451183143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.448909649 +0000 UTC m=+2694.428466489" watchObservedRunningTime="2026-11-03 21:15:01.451183143 +0000 UTC m=+2694.430739984"
Nov 03 21:15:02.349923 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.349893    5318 generic.go:356] "Generic (PLEG): container finished" podID="9437a57f-d923-4cee-bb48-94a536f2964f" containerID="7e18ffe66ed396f142fcd4a0ab5efbd416238470573c17be3ae9043c9eeff725" exitCode=60
Nov 03 21:15:02.350446 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.349973    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerDied","Data":"7e18ffe66ed396f142fcd4a0ab5efbd416238470573c17be3ae9043c9eeff725"}
Nov 03 21:15:02.350446 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.350264    5318 scope.go:122] "RemoveContainer" containerID="7e18ffe66ed396f142fcd4a0ab5efbd416238470573c17be3ae9043c9eeff725"
Nov 03 21:15:02.352329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.352304    5318 generic.go:356] "Generic (PLEG): container finished" podID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerID="6570cc25c600c199ca541eb8a4bec5df9687a7f4a2f0d4ef0d044a4c09da76e1" exitCode=60
Nov 03 21:15:02.352654 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.352629    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" event={"ID":"0ab85640-f691-46b8-8a5d-53143f4afb69","Type":"ContainerDied","Data":"6570cc25c600c199ca541eb8a4bec5df9687a7f4a2f0d4ef0d044a4c09da76e1"}
Nov 03 21:15:02.352792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.352775    5318 scope.go:122] "RemoveContainer" containerID="6570cc25c600c199ca541eb8a4bec5df9687a7f4a2f0d4ef0d044a4c09da76e1"
Nov 03 21:15:02.354554 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.354533    5318 generic.go:356] "Generic (PLEG): container finished" podID="0a81ca46-ec2a-4540-823a-897631b027ae" containerID="7b9813e2728e413bd6457a8aae738771be38b6afd3948710f7c1966f761dd1ce" exitCode=60
Nov 03 21:15:02.354649 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.354625    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" event={"ID":"0a81ca46-ec2a-4540-823a-897631b027ae","Type":"ContainerDied","Data":"7b9813e2728e413bd6457a8aae738771be38b6afd3948710f7c1966f761dd1ce"}
Nov 03 21:15:02.354848 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.354828    5318 scope.go:122] "RemoveContainer" containerID="7b9813e2728e413bd6457a8aae738771be38b6afd3948710f7c1966f761dd1ce"
Nov 03 21:15:02.356557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.356533    5318 generic.go:356] "Generic (PLEG): container finished" podID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerID="102e45f0e66fccb6f8a4edd77a0247262c2d328701ab6bf504417450cc91c005" exitCode=60
Nov 03 21:15:02.356644 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.356567    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" event={"ID":"ce542e76-0469-4ffb-877e-b79f024fbfec","Type":"ContainerDied","Data":"102e45f0e66fccb6f8a4edd77a0247262c2d328701ab6bf504417450cc91c005"}
Nov 03 21:15:02.356830 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.356815    5318 scope.go:122] "RemoveContainer" containerID="102e45f0e66fccb6f8a4edd77a0247262c2d328701ab6bf504417450cc91c005"
Nov 03 21:15:02.521683 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:02.521649    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:03.364386 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.364327    5318 generic.go:356] "Generic (PLEG): container finished" podID="0a81ca46-ec2a-4540-823a-897631b027ae" containerID="0aee63d99e7ac3de43c65435a8a1af00c3210651d6070ffc9add4a24c8edd947" exitCode=60
Nov 03 21:15:03.364819 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.364379    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" event={"ID":"0a81ca46-ec2a-4540-823a-897631b027ae","Type":"ContainerDied","Data":"0aee63d99e7ac3de43c65435a8a1af00c3210651d6070ffc9add4a24c8edd947"}
Nov 03 21:15:03.364819 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.364441    5318 scope.go:122] "RemoveContainer" containerID="7b9813e2728e413bd6457a8aae738771be38b6afd3948710f7c1966f761dd1ce"
Nov 03 21:15:03.364819 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.364730    5318 scope.go:122] "RemoveContainer" containerID="0aee63d99e7ac3de43c65435a8a1af00c3210651d6070ffc9add4a24c8edd947"
Nov 03 21:15:03.364991 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:03.364971    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c_odh-ai-gateway-infra(0a81ca46-ec2a-4540-823a-897631b027ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" podUID="0a81ca46-ec2a-4540-823a-897631b027ae"
Nov 03 21:15:03.368222 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.368120    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" event={"ID":"ce542e76-0469-4ffb-877e-b79f024fbfec","Type":"ContainerStarted","Data":"64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2"}
Nov 03 21:15:03.370492 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.370467    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerStarted","Data":"7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940"}
Nov 03 21:15:03.372529 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.372509    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" event={"ID":"0ab85640-f691-46b8-8a5d-53143f4afb69","Type":"ContainerStarted","Data":"e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5"}
Nov 03 21:15:03.579676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.579626    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" podStartSLOduration=3.579611491 podStartE2EDuration="3.579611491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.578636939 +0000 UTC m=+2696.558193781" watchObservedRunningTime="2026-11-03 21:15:03.579611491 +0000 UTC m=+2696.559168331"
Nov 03 21:15:03.688900 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:03.688852    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" podStartSLOduration=3.688839314 podStartE2EDuration="3.688839314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.686050138 +0000 UTC m=+2696.665606978" watchObservedRunningTime="2026-11-03 21:15:03.688839314 +0000 UTC m=+2696.668396149"
Nov 03 21:15:04.197908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.197878    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"]
Nov 03 21:15:04.198130 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.198102    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy" containerID="cri-o://fae47d652f35ecd8eaf56b054c7e00d7a469cddac5ea9f0b7ebb2e3c14c82135" gracePeriod=30
Nov 03 21:15:04.380605 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.380574    5318 scope.go:122] "RemoveContainer" containerID="0aee63d99e7ac3de43c65435a8a1af00c3210651d6070ffc9add4a24c8edd947"
Nov 03 21:15:04.381094 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:04.380873    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c_odh-ai-gateway-infra(0a81ca46-ec2a-4540-823a-897631b027ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" podUID="0a81ca46-ec2a-4540-823a-897631b027ae"
Nov 03 21:15:04.382218 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.382196    5318 generic.go:356] "Generic (PLEG): container finished" podID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerID="64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2" exitCode=60
Nov 03 21:15:04.382355 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.382290    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" event={"ID":"ce542e76-0469-4ffb-877e-b79f024fbfec","Type":"ContainerDied","Data":"64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2"}
Nov 03 21:15:04.382414 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.382364    5318 scope.go:122] "RemoveContainer" containerID="102e45f0e66fccb6f8a4edd77a0247262c2d328701ab6bf504417450cc91c005"
Nov 03 21:15:04.382476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.382462    5318 scope.go:122] "RemoveContainer" containerID="64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2"
Nov 03 21:15:04.382700 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:04.382674    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n_odh-ai-gateway-infra(ce542e76-0469-4ffb-877e-b79f024fbfec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec"
Nov 03 21:15:04.384159 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.384137    5318 generic.go:356] "Generic (PLEG): container finished" podID="9437a57f-d923-4cee-bb48-94a536f2964f" containerID="7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940" exitCode=60
Nov 03 21:15:04.384244 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.384216    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerDied","Data":"7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940"}
Nov 03 21:15:04.384383 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.384364    5318 scope.go:122] "RemoveContainer" containerID="7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940"
Nov 03 21:15:04.384584 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:04.384564    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn_odh-ai-gateway-infra(9437a57f-d923-4cee-bb48-94a536f2964f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" podUID="9437a57f-d923-4cee-bb48-94a536f2964f"
Nov 03 21:15:04.385978 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.385957    5318 generic.go:356] "Generic (PLEG): container finished" podID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerID="e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5" exitCode=60
Nov 03 21:15:04.386060 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.386013    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" event={"ID":"0ab85640-f691-46b8-8a5d-53143f4afb69","Type":"ContainerDied","Data":"e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5"}
Nov 03 21:15:04.386187 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.386169    5318 scope.go:122] "RemoveContainer" containerID="e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5"
Nov 03 21:15:04.386415 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:04.386398    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s_odh-ai-gateway-infra(0ab85640-f691-46b8-8a5d-53143f4afb69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69"
Nov 03 21:15:04.394464 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.394447    5318 scope.go:122] "RemoveContainer" containerID="7e18ffe66ed396f142fcd4a0ab5efbd416238470573c17be3ae9043c9eeff725"
Nov 03 21:15:04.406802 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:04.406785    5318 scope.go:122] "RemoveContainer" containerID="6570cc25c600c199ca541eb8a4bec5df9687a7f4a2f0d4ef0d044a4c09da76e1"
Nov 03 21:15:05.393617 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:05.393582    5318 scope.go:122] "RemoveContainer" containerID="e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5"
Nov 03 21:15:05.394079 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:05.393870    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s_odh-ai-gateway-infra(0ab85640-f691-46b8-8a5d-53143f4afb69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69"
Nov 03 21:15:05.395231 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:05.395211    5318 scope.go:122] "RemoveContainer" containerID="64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2"
Nov 03 21:15:05.395449 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:05.395426    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n_odh-ai-gateway-infra(ce542e76-0469-4ffb-877e-b79f024fbfec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec"
Nov 03 21:15:05.396801 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:05.396784    5318 scope.go:122] "RemoveContainer" containerID="7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940"
Nov 03 21:15:05.396969 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:05.396952    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn_odh-ai-gateway-infra(9437a57f-d923-4cee-bb48-94a536f2964f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" podUID="9437a57f-d923-4cee-bb48-94a536f2964f"
Nov 03 21:15:07.430667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:07.430621    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:07.714575 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:07.714499    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:15:07.721400 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:07.721377    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:15:08.112263 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:08.112234    5318 scope.go:122] "RemoveContainer" containerID="d1e52d685c629edbee9a48603cd00f2523cba5766f34b891481af88b8622bb72"
Nov 03 21:15:08.122594 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:08.122222    5318 scope.go:122] "RemoveContainer" containerID="04c86674c09aaee0f9756cf739e2a87d029e0b15858381f3396eec771d896fcb"
Nov 03 21:15:09.428321 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.428281    5318 generic.go:356] "Generic (PLEG): container finished" podID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerID="fae47d652f35ecd8eaf56b054c7e00d7a469cddac5ea9f0b7ebb2e3c14c82135" exitCode=0
Nov 03 21:15:09.428720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.428332    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" event={"ID":"1df2746c-2881-4150-875d-e8aacbe6f90b","Type":"ContainerDied","Data":"fae47d652f35ecd8eaf56b054c7e00d7a469cddac5ea9f0b7ebb2e3c14c82135"}
Nov 03 21:15:09.476965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.476897    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 21:15:09.538476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538446    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538512    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538552    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538588    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538616    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538703    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538720    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.538798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538731    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:09.538798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538760    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.539076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538804    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data\") pod \"1df2746c-2881-4150-875d-e8aacbe6f90b\" (UID: \"1df2746c-2881-4150-875d-e8aacbe6f90b\") "
Nov 03 21:15:09.539076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.538859    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:09.539076 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539067    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:09.539238 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539087    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:09.539238 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539165    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:09.539398 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539380    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.539468 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539404    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.539468 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539420    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.539468 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539434    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.539468 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.539448    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1df2746c-2881-4150-875d-e8aacbe6f90b-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.540804 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.540776    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:09.540898 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.540813    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:09.540898 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.540854    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:15:09.540996 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.540979    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw" pod "1df2746c-2881-4150-875d-e8aacbe6f90b" (UID: "1df2746c-2881-4150-875d-e8aacbe6f90b"). InnerVolumeSpecName "kube-api-access-hhpzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:09.640702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.640671    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpzw\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-kube-api-access-hhpzw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.640702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.640700    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.640702 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.640711    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:09.640926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:09.640719    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1df2746c-2881-4150-875d-e8aacbe6f90b-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:10.435612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:10.435584    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"
Nov 03 21:15:10.435612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:10.435600    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh" event={"ID":"1df2746c-2881-4150-875d-e8aacbe6f90b","Type":"ContainerDied","Data":"92bc254f0338ee7472fa79b7fd741f63cd76ab39d02c38770e626337300491d1"}
Nov 03 21:15:10.436119 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:10.435659    5318 scope.go:122] "RemoveContainer" containerID="fae47d652f35ecd8eaf56b054c7e00d7a469cddac5ea9f0b7ebb2e3c14c82135"
Nov 03 21:15:10.641182 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:10.641139    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"]
Nov 03 21:15:10.643939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:10.643913    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-7qkvh"]
Nov 03 21:15:10.747287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:10.747213    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:15:11.256347 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:11.256303    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" path="/var/lib/kubelet/pods/1df2746c-2881-4150-875d-e8aacbe6f90b/volumes"
Nov 03 21:15:12.517164 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:12.517132    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:15.248916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:15.248881    5318 scope.go:122] "RemoveContainer" containerID="64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2"
Nov 03 21:15:16.249196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:16.249165    5318 scope.go:122] "RemoveContainer" containerID="7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940"
Nov 03 21:15:16.249653 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:16.249304    5318 scope.go:122] "RemoveContainer" containerID="0aee63d99e7ac3de43c65435a8a1af00c3210651d6070ffc9add4a24c8edd947"
Nov 03 21:15:16.475424 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:16.475395    5318 generic.go:356] "Generic (PLEG): container finished" podID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerID="00b510496a6096fff207ca87ca0610de9ad3735218731dea570fe31010fa14fa" exitCode=60
Nov 03 21:15:16.475565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:16.475465    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" event={"ID":"ce542e76-0469-4ffb-877e-b79f024fbfec","Type":"ContainerDied","Data":"00b510496a6096fff207ca87ca0610de9ad3735218731dea570fe31010fa14fa"}
Nov 03 21:15:16.475565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:16.475517    5318 scope.go:122] "RemoveContainer" containerID="64224ee2deb31c5415bebe76e684baf96f5e944cf31e44cf8dc947adf7139ee2"
Nov 03 21:15:16.475835 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:16.475820    5318 scope.go:122] "RemoveContainer" containerID="00b510496a6096fff207ca87ca0610de9ad3735218731dea570fe31010fa14fa"
Nov 03 21:15:16.476078 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:16.476060    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n_odh-ai-gateway-infra(ce542e76-0469-4ffb-877e-b79f024fbfec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec"
Nov 03 21:15:17.251636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.251611    5318 scope.go:122] "RemoveContainer" containerID="e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5"
Nov 03 21:15:17.442653 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.442620    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:17.485874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.485842    5318 generic.go:356] "Generic (PLEG): container finished" podID="0a81ca46-ec2a-4540-823a-897631b027ae" containerID="ca6afc1b54dc6d8e459a053ad2d88d27174dddb4fe48eb23d8c2e86903ba0a8d" exitCode=60
Nov 03 21:15:17.486009 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.485915    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" event={"ID":"0a81ca46-ec2a-4540-823a-897631b027ae","Type":"ContainerDied","Data":"ca6afc1b54dc6d8e459a053ad2d88d27174dddb4fe48eb23d8c2e86903ba0a8d"}
Nov 03 21:15:17.486009 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.485987    5318 scope.go:122] "RemoveContainer" containerID="0aee63d99e7ac3de43c65435a8a1af00c3210651d6070ffc9add4a24c8edd947"
Nov 03 21:15:17.486443 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.486404    5318 scope.go:122] "RemoveContainer" containerID="ca6afc1b54dc6d8e459a053ad2d88d27174dddb4fe48eb23d8c2e86903ba0a8d"
Nov 03 21:15:17.487047 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:17.486712    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c_odh-ai-gateway-infra(0a81ca46-ec2a-4540-823a-897631b027ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" podUID="0a81ca46-ec2a-4540-823a-897631b027ae"
Nov 03 21:15:17.490284 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.490256    5318 generic.go:356] "Generic (PLEG): container finished" podID="9437a57f-d923-4cee-bb48-94a536f2964f" containerID="c9a9eb31c867db7e574c33c25f8e45f0a71e160efdf27e1110ff49bc4f2163d8" exitCode=60
Nov 03 21:15:17.490399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.490321    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerDied","Data":"c9a9eb31c867db7e574c33c25f8e45f0a71e160efdf27e1110ff49bc4f2163d8"}
Nov 03 21:15:17.490703 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.490686    5318 scope.go:122] "RemoveContainer" containerID="c9a9eb31c867db7e574c33c25f8e45f0a71e160efdf27e1110ff49bc4f2163d8"
Nov 03 21:15:17.490980 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:17.490955    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn_odh-ai-gateway-infra(9437a57f-d923-4cee-bb48-94a536f2964f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" podUID="9437a57f-d923-4cee-bb48-94a536f2964f"
Nov 03 21:15:17.498846 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.498817    5318 scope.go:122] "RemoveContainer" containerID="7bf59ef57976e8dfac86d4aa9b9b058935daf033523feb000aa0e92b8c2bc940"
Nov 03 21:15:17.595898 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.595866    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"]
Nov 03 21:15:17.747441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.747418    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:17.838615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.838588    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg\" (UniqueName: \"kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg\") pod \"ce542e76-0469-4ffb-877e-b79f024fbfec\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") "
Nov 03 21:15:17.838792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.838714    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca\") pod \"ce542e76-0469-4ffb-877e-b79f024fbfec\" (UID: \"ce542e76-0469-4ffb-877e-b79f024fbfec\") "
Nov 03 21:15:17.839112 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.839092    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca" pod "ce542e76-0469-4ffb-877e-b79f024fbfec" (UID: "ce542e76-0469-4ffb-877e-b79f024fbfec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:17.840729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.840700    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg" pod "ce542e76-0469-4ffb-877e-b79f024fbfec" (UID: "ce542e76-0469-4ffb-877e-b79f024fbfec"). InnerVolumeSpecName "kube-api-access-dfzkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:17.939968 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.939898    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce542e76-0469-4ffb-877e-b79f024fbfec-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:17.939968 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:17.939926    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzkg\" (UniqueName: \"kubernetes.io/projected/ce542e76-0469-4ffb-877e-b79f024fbfec-kube-api-access-dfzkg\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.499311 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.499275    5318 generic.go:356] "Generic (PLEG): container finished" podID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerID="b6f9b6351779132cae967f72ec83a00656c60fd78d2a1c32e8a23536d56d6eb9" exitCode=60
Nov 03 21:15:18.499900 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.499367    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" event={"ID":"0ab85640-f691-46b8-8a5d-53143f4afb69","Type":"ContainerDied","Data":"b6f9b6351779132cae967f72ec83a00656c60fd78d2a1c32e8a23536d56d6eb9"}
Nov 03 21:15:18.499900 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.499421    5318 scope.go:122] "RemoveContainer" containerID="e934978d648c4983fc096b60e6e1f03568a75588727187f0323465a6d5a7e3b5"
Nov 03 21:15:18.499900 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.499799    5318 scope.go:122] "RemoveContainer" containerID="b6f9b6351779132cae967f72ec83a00656c60fd78d2a1c32e8a23536d56d6eb9"
Nov 03 21:15:18.500091 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:18.500053    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s_odh-ai-gateway-infra(0ab85640-f691-46b8-8a5d-53143f4afb69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69"
Nov 03 21:15:18.506344 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.506320    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n" event={"ID":"ce542e76-0469-4ffb-877e-b79f024fbfec","Type":"ContainerDied","Data":"c476b48f4edd6f5855b0f3bd0edd40e94a5d217190ffcee5d6e258bd5832a271"}
Nov 03 21:15:18.506455 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.506321    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"
Nov 03 21:15:18.513250 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.513233    5318 scope.go:122] "RemoveContainer" containerID="00b510496a6096fff207ca87ca0610de9ad3735218731dea570fe31010fa14fa"
Nov 03 21:15:18.609179 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.609144    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"]
Nov 03 21:15:18.769317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.769292    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:18.829413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.829353    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"]
Nov 03 21:15:18.833509 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.833482    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"]
Nov 03 21:15:18.846090 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.846067    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895675-hbx5n"]
Nov 03 21:15:18.849045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.849019    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca\") pod \"0a81ca46-ec2a-4540-823a-897631b027ae\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") "
Nov 03 21:15:18.849174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.849071    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw\" (UniqueName: \"kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw\") pod \"0a81ca46-ec2a-4540-823a-897631b027ae\" (UID: \"0a81ca46-ec2a-4540-823a-897631b027ae\") "
Nov 03 21:15:18.849433 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.849408    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca" pod "0a81ca46-ec2a-4540-823a-897631b027ae" (UID: "0a81ca46-ec2a-4540-823a-897631b027ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:18.853365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.850051    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a81ca46-ec2a-4540-823a-897631b027ae-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.853365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.851521    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw" pod "0a81ca46-ec2a-4540-823a-897631b027ae" (UID: "0a81ca46-ec2a-4540-823a-897631b027ae"). InnerVolumeSpecName "kube-api-access-hnfnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:18.951646 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.951619    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnfnw\" (UniqueName: \"kubernetes.io/projected/0a81ca46-ec2a-4540-823a-897631b027ae-kube-api-access-hnfnw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:18.990092 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:18.990068    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:19.154247 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.154216    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca\") pod \"9437a57f-d923-4cee-bb48-94a536f2964f\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") "
Nov 03 21:15:19.154460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.154322    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c\" (UniqueName: \"kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c\") pod \"9437a57f-d923-4cee-bb48-94a536f2964f\" (UID: \"9437a57f-d923-4cee-bb48-94a536f2964f\") "
Nov 03 21:15:19.154664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.154641    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca" pod "9437a57f-d923-4cee-bb48-94a536f2964f" (UID: "9437a57f-d923-4cee-bb48-94a536f2964f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:19.154818 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.154799    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9437a57f-d923-4cee-bb48-94a536f2964f-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.156442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.156422    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c" pod "9437a57f-d923-4cee-bb48-94a536f2964f" (UID: "9437a57f-d923-4cee-bb48-94a536f2964f"). InnerVolumeSpecName "kube-api-access-x6f9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:19.255759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.255724    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" path="/var/lib/kubelet/pods/ce542e76-0469-4ffb-877e-b79f024fbfec/volumes"
Nov 03 21:15:19.256078 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.256041    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6f9c\" (UniqueName: \"kubernetes.io/projected/9437a57f-d923-4cee-bb48-94a536f2964f-kube-api-access-x6f9c\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.516257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.516187    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"
Nov 03 21:15:19.516257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.516194    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c" event={"ID":"0a81ca46-ec2a-4540-823a-897631b027ae","Type":"ContainerDied","Data":"60611cf2b4d46fa8fdc9cdb9070439e00a7eb136c7141d75a4b6a474a595e892"}
Nov 03 21:15:19.516751 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.516261    5318 scope.go:122] "RemoveContainer" containerID="ca6afc1b54dc6d8e459a053ad2d88d27174dddb4fe48eb23d8c2e86903ba0a8d"
Nov 03 21:15:19.519198 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.519175    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn" event={"ID":"9437a57f-d923-4cee-bb48-94a536f2964f","Type":"ContainerDied","Data":"0c35816e6fb3dc335b4bb568eb813b393b458266fae21b09017eeb347f2caa68"}
Nov 03 21:15:19.519393 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.519200    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"
Nov 03 21:15:19.527111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.527089    5318 scope.go:122] "RemoveContainer" containerID="c9a9eb31c867db7e574c33c25f8e45f0a71e160efdf27e1110ff49bc4f2163d8"
Nov 03 21:15:19.622442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.622417    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"]
Nov 03 21:15:19.671633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.671596    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"]
Nov 03 21:15:19.677958 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.677931    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1f1855-29895675-9ng9c"]
Nov 03 21:15:19.744005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.743981    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:19.779379 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.779290    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"]
Nov 03 21:15:19.784633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.784609    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c971dc-29895675-lxcgn"]
Nov 03 21:15:19.860925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.860895    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk\" (UniqueName: \"kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk\") pod \"0ab85640-f691-46b8-8a5d-53143f4afb69\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") "
Nov 03 21:15:19.861051 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.860971    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca\") pod \"0ab85640-f691-46b8-8a5d-53143f4afb69\" (UID: \"0ab85640-f691-46b8-8a5d-53143f4afb69\") "
Nov 03 21:15:19.861350 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.861317    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca" pod "0ab85640-f691-46b8-8a5d-53143f4afb69" (UID: "0ab85640-f691-46b8-8a5d-53143f4afb69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:19.863010 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.862988    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk" pod "0ab85640-f691-46b8-8a5d-53143f4afb69" (UID: "0ab85640-f691-46b8-8a5d-53143f4afb69"). InnerVolumeSpecName "kube-api-access-h2mdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:19.962797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.962767    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2mdk\" (UniqueName: \"kubernetes.io/projected/0ab85640-f691-46b8-8a5d-53143f4afb69-kube-api-access-h2mdk\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:19.962797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:19.962795    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab85640-f691-46b8-8a5d-53143f4afb69-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:20.528016 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:20.527984    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s" event={"ID":"0ab85640-f691-46b8-8a5d-53143f4afb69","Type":"ContainerDied","Data":"b27125d963ff20742c3c8b02b6c881b678b92a558a6c22c236b60e5c0bf697a3"}
Nov 03 21:15:20.528016 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:20.528007    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"
Nov 03 21:15:20.528505 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:20.528039    5318 scope.go:122] "RemoveContainer" containerID="b6f9b6351779132cae967f72ec83a00656c60fd78d2a1c32e8a23536d56d6eb9"
Nov 03 21:15:20.740615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:20.740584    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"]
Nov 03 21:15:20.786052 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:20.785979    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-df71d7-29895675-vgz5s"]
Nov 03 21:15:21.022567 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:21.022526    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:21.256889 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:21.256854    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" path="/var/lib/kubelet/pods/0a81ca46-ec2a-4540-823a-897631b027ae/volumes"
Nov 03 21:15:21.257275 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:21.257260    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" path="/var/lib/kubelet/pods/0ab85640-f691-46b8-8a5d-53143f4afb69/volumes"
Nov 03 21:15:21.257673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:21.257658    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" path="/var/lib/kubelet/pods/9437a57f-d923-4cee-bb48-94a536f2964f/volumes"
Nov 03 21:15:22.430222 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:22.430187    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:25.933969 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:25.933932    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:15:27.308664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:27.308628    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:28.553657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:28.553623    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:32.203858 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:32.203828    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:35.429805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:35.429770    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:35.637681 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:35.637651    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:36.835447 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:36.835410    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"]
Nov 03 21:15:36.919842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:36.919708    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:37.149717 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:37.149686    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:41.297272 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.297231    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"]
Nov 03 21:15:41.297794 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.297496    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" podUID="acea590a-9071-4341-8096-c1b68b2788c1" containerName="maas-api" containerID="cri-o://74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a" gracePeriod=30
Nov 03 21:15:41.466852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.466815    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:41.621693 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.621663    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:15:41.669686 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.669655    5318 generic.go:356] "Generic (PLEG): container finished" podID="acea590a-9071-4341-8096-c1b68b2788c1" containerID="74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a" exitCode=0
Nov 03 21:15:41.669842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.669697    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"
Nov 03 21:15:41.669842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.669734    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" event={"ID":"acea590a-9071-4341-8096-c1b68b2788c1","Type":"ContainerDied","Data":"74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a"}
Nov 03 21:15:41.669842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.669791    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k" event={"ID":"acea590a-9071-4341-8096-c1b68b2788c1","Type":"ContainerDied","Data":"1341eac3562378e53b4b09abdcdd5b6cc8daf7e8e1064aa27dbde60680b1a33f"}
Nov 03 21:15:41.669842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.669822    5318 scope.go:122] "RemoveContainer" containerID="74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a"
Nov 03 21:15:41.681190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.681072    5318 scope.go:122] "RemoveContainer" containerID="74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a"
Nov 03 21:15:41.681750 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:41.681677    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a\": container with ID starting with 74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a not found: ID does not exist" containerID="74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a"
Nov 03 21:15:41.681750 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.681718    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a"} err="failed to get container status \"74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a\": rpc error: code = NotFound desc = could not find container \"74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a\": container with ID starting with 74130a6684e61f4d8b53457ba7276962435b95f6b3bd20d17bd9f612862c4f7a not found: ID does not exist"
Nov 03 21:15:41.723827 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.723795    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca\") pod \"acea590a-9071-4341-8096-c1b68b2788c1\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") "
Nov 03 21:15:41.724022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.723935    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5\" (UniqueName: \"kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5\") pod \"acea590a-9071-4341-8096-c1b68b2788c1\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") "
Nov 03 21:15:41.724022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.723968    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\") pod \"acea590a-9071-4341-8096-c1b68b2788c1\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") "
Nov 03 21:15:41.724157 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.724090    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs\") pod \"acea590a-9071-4341-8096-c1b68b2788c1\" (UID: \"acea590a-9071-4341-8096-c1b68b2788c1\") "
Nov 03 21:15:41.724219 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.724148    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca" pod "acea590a-9071-4341-8096-c1b68b2788c1" (UID: "acea590a-9071-4341-8096-c1b68b2788c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:41.724538 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.724516    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea590a-9071-4341-8096-c1b68b2788c1-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:41.726352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.726315    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls" pod "acea590a-9071-4341-8096-c1b68b2788c1" (UID: "acea590a-9071-4341-8096-c1b68b2788c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:41.726537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.726515    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs" pod "acea590a-9071-4341-8096-c1b68b2788c1" (UID: "acea590a-9071-4341-8096-c1b68b2788c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:41.728749 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.728725    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5" pod "acea590a-9071-4341-8096-c1b68b2788c1" (UID: "acea590a-9071-4341-8096-c1b68b2788c1"). InnerVolumeSpecName "kube-api-access-x69n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:41.825318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.825287    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:41.825318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.825315    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x69n5\" (UniqueName: \"kubernetes.io/projected/acea590a-9071-4341-8096-c1b68b2788c1-kube-api-access-x69n5\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:41.825318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:41.825326    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea590a-9071-4341-8096-c1b68b2788c1-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:42.082639 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:42.082601    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"]
Nov 03 21:15:42.087069 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:42.087041    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10fa59-7c4ff9965b-9t66k"]
Nov 03 21:15:43.023984 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:43.023957    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:43.241217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:43.241188    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:43.255551 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:43.255526    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acea590a-9071-4341-8096-c1b68b2788c1" path="/var/lib/kubelet/pods/acea590a-9071-4341-8096-c1b68b2788c1/volumes"
Nov 03 21:15:43.441705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:43.441667    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:44.047948 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:44.047913    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:44.052219 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:44.052189    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:15:48.532294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:48.532260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:50.640464 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:50.640433    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:50.858397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:50.852418    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:51.058161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.058075    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:51.408351 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.408308    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:51.857619 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.857579    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g" podUID="de86dad5-1f13-46a5-87f0-21cb0452229e" containerName="main" containerID="cri-o://02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a" gracePeriod=45
Nov 03 21:15:51.878288 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878258    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"]
Nov 03 21:15:51.878759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878746    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.878799 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878761    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.878799 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878774    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.878799 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878779    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.878799 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878792    5318 cpu_manager.go:414] "containerMap: removing container" podUID="acea590a-9071-4341-8096-c1b68b2788c1" containerName="maas-api"
Nov 03 21:15:51.878799 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878797    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acea590a-9071-4341-8096-c1b68b2788c1" containerName="maas-api"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878805    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878810    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878816    5318 cpu_manager.go:414] "containerMap: removing container" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878821    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878827    5318 cpu_manager.go:414] "containerMap: removing container" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878832    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878840    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878845    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878856    5318 cpu_manager.go:414] "containerMap: removing container" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878861    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878873    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878878    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878886    5318 cpu_manager.go:414] "containerMap: removing container" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878892    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878905    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878911    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878929    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878938    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878951    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878958    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878967    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.878999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.878971    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879081    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="acea590a-9071-4341-8096-c1b68b2788c1" containerName="maas-api"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879097    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879106    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879114    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879123    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879129    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce542e76-0469-4ffb-877e-b79f024fbfec" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879137    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879144    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df2746c-2881-4150-875d-e8aacbe6f90b" containerName="istio-proxy"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879152    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879161    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879171    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879178    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab85640-f691-46b8-8a5d-53143f4afb69" containerName="cleanup"
Nov 03 21:15:51.879797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.879191    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a81ca46-ec2a-4540-823a-897631b027ae" containerName="cleanup"
Nov 03 21:15:51.882862 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.882843    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:51.931483 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.931453    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:15:51.931483 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.931462    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:15:51.963775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.963730    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srq7\" (UniqueName: \"kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:51.963965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.963913    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:51.963965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.963944    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:51.981545 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:51.981514    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"]
Nov 03 21:15:52.066010 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.065972    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srq7\" (UniqueName: \"kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.066200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.066073    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.066200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.066106    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.067703 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.067670    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.073765 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.073734    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.124022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.123952    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srq7\" (UniqueName: \"kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7\") pod \"maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.127361 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.127329    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:15:52.167317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167274    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home\") pod \"de86dad5-1f13-46a5-87f0-21cb0452229e\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") "
Nov 03 21:15:52.167317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167319    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir\") pod \"de86dad5-1f13-46a5-87f0-21cb0452229e\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") "
Nov 03 21:15:52.167604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167376    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd\" (UniqueName: \"kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd\") pod \"de86dad5-1f13-46a5-87f0-21cb0452229e\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") "
Nov 03 21:15:52.167604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167441    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache\") pod \"de86dad5-1f13-46a5-87f0-21cb0452229e\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") "
Nov 03 21:15:52.167604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167500    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs\" (UniqueName: \"kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs\") pod \"de86dad5-1f13-46a5-87f0-21cb0452229e\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") "
Nov 03 21:15:52.167775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167636    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm\") pod \"de86dad5-1f13-46a5-87f0-21cb0452229e\" (UID: \"de86dad5-1f13-46a5-87f0-21cb0452229e\") "
Nov 03 21:15:52.167775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167635    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home" pod "de86dad5-1f13-46a5-87f0-21cb0452229e" (UID: "de86dad5-1f13-46a5-87f0-21cb0452229e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:52.167902 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.167824    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir" pod "de86dad5-1f13-46a5-87f0-21cb0452229e" (UID: "de86dad5-1f13-46a5-87f0-21cb0452229e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:52.168180 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.168155    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache" pod "de86dad5-1f13-46a5-87f0-21cb0452229e" (UID: "de86dad5-1f13-46a5-87f0-21cb0452229e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:52.168350 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.168302    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.168350 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.168322    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.168525 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.168358    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.170376 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.170313    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs" pod "de86dad5-1f13-46a5-87f0-21cb0452229e" (UID: "de86dad5-1f13-46a5-87f0-21cb0452229e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:52.170376 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.170327    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd" pod "de86dad5-1f13-46a5-87f0-21cb0452229e" (UID: "de86dad5-1f13-46a5-87f0-21cb0452229e"). InnerVolumeSpecName "kube-api-access-xs2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:52.170376 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.170329    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm" pod "de86dad5-1f13-46a5-87f0-21cb0452229e" (UID: "de86dad5-1f13-46a5-87f0-21cb0452229e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:52.242213 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.242177    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:52.269788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.269747    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2fd\" (UniqueName: \"kubernetes.io/projected/de86dad5-1f13-46a5-87f0-21cb0452229e-kube-api-access-xs2fd\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.269997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.269975    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86dad5-1f13-46a5-87f0-21cb0452229e-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.270083 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.270005    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de86dad5-1f13-46a5-87f0-21cb0452229e-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:52.313760 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.311254    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"]
Nov 03 21:15:52.416415 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.416193    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:52.640555 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.640533    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"]
Nov 03 21:15:52.643272 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:15:52.643242    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be203ef_5dd1_43b6_9430_9bf364b73d84.slice/crio-2523d48853eb11ed3771ffa46204a3526e4141b06f448633acb44cbf633c6657 WatchSource:0}: Error finding container 2523d48853eb11ed3771ffa46204a3526e4141b06f448633acb44cbf633c6657: Status 404 returned error can't find the container with id 2523d48853eb11ed3771ffa46204a3526e4141b06f448633acb44cbf633c6657
Nov 03 21:15:52.745603 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.745568    5318 generic.go:356] "Generic (PLEG): container finished" podID="de86dad5-1f13-46a5-87f0-21cb0452229e" containerID="02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a" exitCode=0
Nov 03 21:15:52.745603 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.745608    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"
Nov 03 21:15:52.745864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.745651    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g" event={"ID":"de86dad5-1f13-46a5-87f0-21cb0452229e","Type":"ContainerDied","Data":"02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a"}
Nov 03 21:15:52.745864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.745691    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g" event={"ID":"de86dad5-1f13-46a5-87f0-21cb0452229e","Type":"ContainerDied","Data":"04bd511c652904ba5a8ab37a2421fcc992d34febe43a8c7e6f4d56cafb5d9003"}
Nov 03 21:15:52.745864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.745710    5318 scope.go:122] "RemoveContainer" containerID="02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a"
Nov 03 21:15:52.747370 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.747324    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8" event={"ID":"5be203ef-5dd1-43b6-9430-9bf364b73d84","Type":"ContainerStarted","Data":"2523d48853eb11ed3771ffa46204a3526e4141b06f448633acb44cbf633c6657"}
Nov 03 21:15:52.762986 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.762767    5318 scope.go:122] "RemoveContainer" containerID="02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a"
Nov 03 21:15:52.763267 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:52.763208    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a\": container with ID starting with 02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a not found: ID does not exist" containerID="02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a"
Nov 03 21:15:52.763267 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.763241    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a"} err="failed to get container status \"02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a\": rpc error: code = NotFound desc = could not find container \"02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a\": container with ID starting with 02f9ba75fb2baa22eff69a896940f5899183c8f14f78078a9d6fb5306419a00a not found: ID does not exist"
Nov 03 21:15:52.900620 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.900593    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"]
Nov 03 21:15:52.905010 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:52.904986    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-df71d7/test-model-df71d7-kserve-b7ff8fd69-tz99g"]
Nov 03 21:15:53.255984 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:53.255912    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de86dad5-1f13-46a5-87f0-21cb0452229e" path="/var/lib/kubelet/pods/de86dad5-1f13-46a5-87f0-21cb0452229e/volumes"
Nov 03 21:15:53.723305 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:53.723265    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:53.755161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:53.755128    5318 generic.go:356] "Generic (PLEG): container finished" podID="5be203ef-5dd1-43b6-9430-9bf364b73d84" containerID="5f99a947831264cfc981cccf770e8aeccb4733db1a5768b2696d5b455d656857" exitCode=0
Nov 03 21:15:53.755356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:53.755209    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8" event={"ID":"5be203ef-5dd1-43b6-9430-9bf364b73d84","Type":"ContainerDied","Data":"5f99a947831264cfc981cccf770e8aeccb4733db1a5768b2696d5b455d656857"}
Nov 03 21:15:54.898789 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:54.898762    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:55.002670 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.002637    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca\") pod \"5be203ef-5dd1-43b6-9430-9bf364b73d84\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") "
Nov 03 21:15:55.002874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.002699    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7\" (UniqueName: \"kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7\") pod \"5be203ef-5dd1-43b6-9430-9bf364b73d84\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") "
Nov 03 21:15:55.002874 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.002859    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca\") pod \"5be203ef-5dd1-43b6-9430-9bf364b73d84\" (UID: \"5be203ef-5dd1-43b6-9430-9bf364b73d84\") "
Nov 03 21:15:55.003518 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.003488    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca" pod "5be203ef-5dd1-43b6-9430-9bf364b73d84" (UID: "5be203ef-5dd1-43b6-9430-9bf364b73d84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:55.005176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.005150    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7" pod "5be203ef-5dd1-43b6-9430-9bf364b73d84" (UID: "5be203ef-5dd1-43b6-9430-9bf364b73d84"). InnerVolumeSpecName "kube-api-access-4srq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:55.005364 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.005325    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca" pod "5be203ef-5dd1-43b6-9430-9bf364b73d84" (UID: "5be203ef-5dd1-43b6-9430-9bf364b73d84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:55.104502 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.104454    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.104502 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.104495    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be203ef-5dd1-43b6-9430-9bf364b73d84-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.104502 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.104510    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srq7\" (UniqueName: \"kubernetes.io/projected/5be203ef-5dd1-43b6-9430-9bf364b73d84-kube-api-access-4srq7\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.769496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.769464    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"
Nov 03 21:15:55.769671 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.769509    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8" event={"ID":"5be203ef-5dd1-43b6-9430-9bf364b73d84","Type":"ContainerDied","Data":"2523d48853eb11ed3771ffa46204a3526e4141b06f448633acb44cbf633c6657"}
Nov 03 21:15:55.769671 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:55.769599    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2523d48853eb11ed3771ffa46204a3526e4141b06f448633acb44cbf633c6657"
Nov 03 21:15:55.934508 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:15:55.934469    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:15:56.277433 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:56.277402    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:56.329371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:56.329344    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"]
Nov 03 21:15:56.483092 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:56.483057    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:56.686733 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:56.686694    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:15:58.947332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:15:58.947262    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:03.424919 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:03.424877    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"]
Nov 03 21:16:03.537005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:03.536967    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"]
Nov 03 21:16:03.646413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:03.646377    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:03.856189 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:03.856158    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:04.064489 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:04.064454    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:05.305534 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:05.305488    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:05.531674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:05.531633    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:05.752071 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:05.752034    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:06.237886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:06.237858    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:06.439757 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:06.439722    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:06.644177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:06.644149    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:07.351162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.351115    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q" podUID="1b8db9e4-0ca4-4d87-8789-271ea2601896" containerName="main" containerID="cri-o://26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910" gracePeriod=45
Nov 03 21:16:07.627628 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.627602    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:16:07.752805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.752767    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm\") pod \"1b8db9e4-0ca4-4d87-8789-271ea2601896\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") "
Nov 03 21:16:07.752970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.752826    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir\") pod \"1b8db9e4-0ca4-4d87-8789-271ea2601896\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") "
Nov 03 21:16:07.752970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.752865    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs\") pod \"1b8db9e4-0ca4-4d87-8789-271ea2601896\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") "
Nov 03 21:16:07.752970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.752916    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66\" (UniqueName: \"kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66\") pod \"1b8db9e4-0ca4-4d87-8789-271ea2601896\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") "
Nov 03 21:16:07.753162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.752977    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache\") pod \"1b8db9e4-0ca4-4d87-8789-271ea2601896\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") "
Nov 03 21:16:07.753162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.753007    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home\") pod \"1b8db9e4-0ca4-4d87-8789-271ea2601896\" (UID: \"1b8db9e4-0ca4-4d87-8789-271ea2601896\") "
Nov 03 21:16:07.753413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.753384    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache" pod "1b8db9e4-0ca4-4d87-8789-271ea2601896" (UID: "1b8db9e4-0ca4-4d87-8789-271ea2601896"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.753623 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.753602    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.753795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.753734    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir" pod "1b8db9e4-0ca4-4d87-8789-271ea2601896" (UID: "1b8db9e4-0ca4-4d87-8789-271ea2601896"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.754227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.754190    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home" pod "1b8db9e4-0ca4-4d87-8789-271ea2601896" (UID: "1b8db9e4-0ca4-4d87-8789-271ea2601896"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.756247 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.756222    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm" pod "1b8db9e4-0ca4-4d87-8789-271ea2601896" (UID: "1b8db9e4-0ca4-4d87-8789-271ea2601896"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:07.756329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.756267    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66" pod "1b8db9e4-0ca4-4d87-8789-271ea2601896" (UID: "1b8db9e4-0ca4-4d87-8789-271ea2601896"). InnerVolumeSpecName "kube-api-access-r2n66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:07.756547 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.756528    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs" pod "1b8db9e4-0ca4-4d87-8789-271ea2601896" (UID: "1b8db9e4-0ca4-4d87-8789-271ea2601896"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:07.847154 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.847085    5318 generic.go:356] "Generic (PLEG): container finished" podID="1b8db9e4-0ca4-4d87-8789-271ea2601896" containerID="26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910" exitCode=0
Nov 03 21:16:07.847461 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.847249    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q" event={"ID":"1b8db9e4-0ca4-4d87-8789-271ea2601896","Type":"ContainerDied","Data":"26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910"}
Nov 03 21:16:07.847600 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.847491    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q" event={"ID":"1b8db9e4-0ca4-4d87-8789-271ea2601896","Type":"ContainerDied","Data":"84d4a9d400b18eb0a411150d7a56da3a15264d7f625a109b02f56d9f3e98f378"}
Nov 03 21:16:07.847600 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.847518    5318 scope.go:122] "RemoveContainer" containerID="26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910"
Nov 03 21:16:07.847600 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.847360    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"
Nov 03 21:16:07.855279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.855213    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.855279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.855242    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.855279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.855257    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8db9e4-0ca4-4d87-8789-271ea2601896-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.855279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.855271    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2n66\" (UniqueName: \"kubernetes.io/projected/1b8db9e4-0ca4-4d87-8789-271ea2601896-kube-api-access-r2n66\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.855793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.855286    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8db9e4-0ca4-4d87-8789-271ea2601896-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:07.862560 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.862535    5318 scope.go:122] "RemoveContainer" containerID="26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910"
Nov 03 21:16:07.862877 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:07.862857    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910\": container with ID starting with 26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910 not found: ID does not exist" containerID="26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910"
Nov 03 21:16:07.862967 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:07.862889    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910"} err="failed to get container status \"26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910\": rpc error: code = NotFound desc = could not find container \"26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910\": container with ID starting with 26acef7374d8e4a51fdaa9b697f72b26d2bba689ead304b124e6663394ed6910 not found: ID does not exist"
Nov 03 21:16:08.001775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.001744    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"]
Nov 03 21:16:08.007495 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.007469    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fc4e8d/test-model-fc4e8d-kserve-f47d6fdf5-wtt6q"]
Nov 03 21:16:08.166833 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.166765    5318 scope.go:122] "RemoveContainer" containerID="8c65f6f4661f9104e449ca243f407b87076841e1c55faf9dae91904c30ffd9de"
Nov 03 21:16:08.426062 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.425962    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2" podUID="d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" containerName="payload-processing" containerID="cri-o://4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4" gracePeriod=25
Nov 03 21:16:08.539308 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.539273    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerName="payload-pre-processing" containerID="cri-o://3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e" gracePeriod=25
Nov 03 21:16:08.682682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.682630    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:16:08.763528 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.763502    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh\" (UniqueName: \"kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh\") pod \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") "
Nov 03 21:16:08.763655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.763556    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume\") pod \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\" (UID: \"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec\") "
Nov 03 21:16:08.763979 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.763948    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume" pod "d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" (UID: "d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:08.765488 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.765470    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh" pod "d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" (UID: "d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec"). InnerVolumeSpecName "kube-api-access-9wqgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:08.854038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.854004    5318 generic.go:356] "Generic (PLEG): container finished" podID="d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" containerID="4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4" exitCode=0
Nov 03 21:16:08.854172 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.854050    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"
Nov 03 21:16:08.854172 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.854091    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2" event={"ID":"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec","Type":"ContainerDied","Data":"4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4"}
Nov 03 21:16:08.854172 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.854145    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2" event={"ID":"d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec","Type":"ContainerDied","Data":"4937281b20f1be05f4f96406c76ef650d2fe98d6372a48e3be8812fd13d48913"}
Nov 03 21:16:08.854287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.854184    5318 scope.go:122] "RemoveContainer" containerID="4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4"
Nov 03 21:16:08.863585 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.863569    5318 scope.go:122] "RemoveContainer" containerID="4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4"
Nov 03 21:16:08.863839 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:08.863813    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4\": container with ID starting with 4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4 not found: ID does not exist" containerID="4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4"
Nov 03 21:16:08.863904 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.863845    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4"} err="failed to get container status \"4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4\": rpc error: code = NotFound desc = could not find container \"4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4\": container with ID starting with 4bae63449518662e687fe748298e3fc8f1ba0bc59498c219a6f1371d542224f4 not found: ID does not exist"
Nov 03 21:16:08.865537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.865517    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqgh\" (UniqueName: \"kubernetes.io/projected/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-kube-api-access-9wqgh\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:08.865630 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:08.865541    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:09.006098 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:09.006008    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:09.007788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:09.007761    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"]
Nov 03 21:16:09.011621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:09.011596    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-df71d7-56b686dc5f-gs2s2"]
Nov 03 21:16:09.254893 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:09.254862    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8db9e4-0ca4-4d87-8789-271ea2601896" path="/var/lib/kubelet/pods/1b8db9e4-0ca4-4d87-8789-271ea2601896/volumes"
Nov 03 21:16:09.255239 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:09.255226    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" path="/var/lib/kubelet/pods/d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec/volumes"
Nov 03 21:16:10.301143 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301056    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"]
Nov 03 21:16:10.301618 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301603    5318 cpu_manager.go:414] "containerMap: removing container" podUID="d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" containerName="payload-processing"
Nov 03 21:16:10.301663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301619    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" containerName="payload-processing"
Nov 03 21:16:10.301663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301640    5318 cpu_manager.go:414] "containerMap: removing container" podUID="1b8db9e4-0ca4-4d87-8789-271ea2601896" containerName="main"
Nov 03 21:16:10.301663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301645    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8db9e4-0ca4-4d87-8789-271ea2601896" containerName="main"
Nov 03 21:16:10.301663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301659    5318 cpu_manager.go:414] "containerMap: removing container" podUID="5be203ef-5dd1-43b6-9430-9bf364b73d84" containerName="revoke-keys"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301665    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be203ef-5dd1-43b6-9430-9bf364b73d84" containerName="revoke-keys"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301677    5318 cpu_manager.go:414] "containerMap: removing container" podUID="de86dad5-1f13-46a5-87f0-21cb0452229e" containerName="main"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301683    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de86dad5-1f13-46a5-87f0-21cb0452229e" containerName="main"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301748    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9437a57f-d923-4cee-bb48-94a536f2964f" containerName="cleanup"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301755    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8db9e4-0ca4-4d87-8789-271ea2601896" containerName="main"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301764    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be203ef-5dd1-43b6-9430-9bf364b73d84" containerName="revoke-keys"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301772    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="de86dad5-1f13-46a5-87f0-21cb0452229e" containerName="main"
Nov 03 21:16:10.301787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.301781    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e99ba1-7cab-4ebc-b6f5-0ebd07fac0ec" containerName="payload-processing"
Nov 03 21:16:10.306358 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.306319    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.357986 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.357960    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:16:10.404129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.404098    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:16:10.406640 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.406615    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"]
Nov 03 21:16:10.483216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.483180    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.483381 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.483352    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw65c\" (UniqueName: \"kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.483442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.483412    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.584308 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.584238    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.584471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.584332    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw65c\" (UniqueName: \"kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.584471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.584386    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.585174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.585147    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.586655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.586633    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.640737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.640712    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw65c\" (UniqueName: \"kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c\") pod \"maas-api-revoke-sub-7add005a-8qm2z\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:10.919887 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:10.919857    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:11.232870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.232783    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:11.290652 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.290622    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"]
Nov 03 21:16:11.295131 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:16:11.295101    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21904997_954f_4cba_9290_50844e8d5561.slice/crio-159f5d128ab46728be442ba31d114677b9b5e47bbc74e534fea84914fb2eab95 WatchSource:0}: Error finding container 159f5d128ab46728be442ba31d114677b9b5e47bbc74e534fea84914fb2eab95: Status 404 returned error can't find the container with id 159f5d128ab46728be442ba31d114677b9b5e47bbc74e534fea84914fb2eab95
Nov 03 21:16:11.349721 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.349693    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main" containerID="cri-o://11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65" gracePeriod=45
Nov 03 21:16:11.737871 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.737844    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.878255    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z" event={"ID":"21904997-954f-4cba-9290-50844e8d5561","Type":"ContainerStarted","Data":"098022850b0cbc2ffb561cb75761e44f761aed9ac1e42004f7e10cb04d3a5c9a"}
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.878399    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z" event={"ID":"21904997-954f-4cba-9290-50844e8d5561","Type":"ContainerStarted","Data":"159f5d128ab46728be442ba31d114677b9b5e47bbc74e534fea84914fb2eab95"}
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.879885    5318 generic.go:356] "Generic (PLEG): container finished" podID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerID="11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65" exitCode=0
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.880008    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" event={"ID":"8fc35bbb-790d-42f9-9ffb-623c5761b34a","Type":"ContainerDied","Data":"11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65"}
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.880039    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw" event={"ID":"8fc35bbb-790d-42f9-9ffb-623c5761b34a","Type":"ContainerDied","Data":"b5dc65fd0821922e84b12a8c5b553a2fa0eef1d426e643d59f94caeeb5b2c8db"}
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.880059    5318 scope.go:122] "RemoveContainer" containerID="11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65"
Nov 03 21:16:11.881357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.880183    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"
Nov 03 21:16:11.893404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.893383    5318 scope.go:122] "RemoveContainer" containerID="11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65"
Nov 03 21:16:11.895347 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:11.895299    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65\": container with ID starting with 11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65 not found: ID does not exist" containerID="11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65"
Nov 03 21:16:11.895475 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.895360    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65"} err="failed to get container status \"11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65\": rpc error: code = NotFound desc = could not find container \"11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65\": container with ID starting with 11693b7b5978623a4bcd54a13fedd41f8a52ca5b970a1fa3a9fdc7a5aeed9d65 not found: ID does not exist"
Nov 03 21:16:11.906300 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.906272    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf\" (UniqueName: \"kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf\") pod \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") "
Nov 03 21:16:11.906433 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.906322    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm\") pod \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") "
Nov 03 21:16:11.906433 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.906379    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home\") pod \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") "
Nov 03 21:16:11.906550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.906444    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache\") pod \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") "
Nov 03 21:16:11.906550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.906503    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs\") pod \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") "
Nov 03 21:16:11.906689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.906579    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir\") pod \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\" (UID: \"8fc35bbb-790d-42f9-9ffb-623c5761b34a\") "
Nov 03 21:16:11.907206 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.907179    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir" pod "8fc35bbb-790d-42f9-9ffb-623c5761b34a" (UID: "8fc35bbb-790d-42f9-9ffb-623c5761b34a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.907303 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.907279    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache" pod "8fc35bbb-790d-42f9-9ffb-623c5761b34a" (UID: "8fc35bbb-790d-42f9-9ffb-623c5761b34a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.907392 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.907373    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home" pod "8fc35bbb-790d-42f9-9ffb-623c5761b34a" (UID: "8fc35bbb-790d-42f9-9ffb-623c5761b34a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.909613 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.909586    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm" pod "8fc35bbb-790d-42f9-9ffb-623c5761b34a" (UID: "8fc35bbb-790d-42f9-9ffb-623c5761b34a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:11.909719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.909658    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs" pod "8fc35bbb-790d-42f9-9ffb-623c5761b34a" (UID: "8fc35bbb-790d-42f9-9ffb-623c5761b34a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:11.915753 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.915725    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf" pod "8fc35bbb-790d-42f9-9ffb-623c5761b34a" (UID: "8fc35bbb-790d-42f9-9ffb-623c5761b34a"). InnerVolumeSpecName "kube-api-access-9phvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:11.987190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:11.987135    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z" podStartSLOduration=1.987120242 podStartE2EDuration="1.987120242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:11.985816681 +0000 UTC m=+2764.965373523" watchObservedRunningTime="2026-11-03 21:16:11.987120242 +0000 UTC m=+2764.966677082"
Nov 03 21:16:12.007824 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.007792    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.008004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.007837    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.008004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.007851    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.008004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.007864    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.008004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.007877    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9phvf\" (UniqueName: \"kubernetes.io/projected/8fc35bbb-790d-42f9-9ffb-623c5761b34a-kube-api-access-9phvf\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.008004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.007890    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc35bbb-790d-42f9-9ffb-623c5761b34a-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:12.293345 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.293314    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"]
Nov 03 21:16:12.298991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.298961    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3b8feb/simulator-w2-3b8feb-kserve-7c5665bd4f-t6tgw"]
Nov 03 21:16:12.888536 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.888448    5318 generic.go:356] "Generic (PLEG): container finished" podID="21904997-954f-4cba-9290-50844e8d5561" containerID="098022850b0cbc2ffb561cb75761e44f761aed9ac1e42004f7e10cb04d3a5c9a" exitCode=0
Nov 03 21:16:12.888536 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:12.888518    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z" event={"ID":"21904997-954f-4cba-9290-50844e8d5561","Type":"ContainerDied","Data":"098022850b0cbc2ffb561cb75761e44f761aed9ac1e42004f7e10cb04d3a5c9a"}
Nov 03 21:16:13.254838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:13.254756    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" path="/var/lib/kubelet/pods/8fc35bbb-790d-42f9-9ffb-623c5761b34a/volumes"
Nov 03 21:16:13.462690 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:13.462643    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:13.688435 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:13.688398    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:13.904561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:13.904533    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:14.063557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.063532    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:14.129448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.129418    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca\") pod \"21904997-954f-4cba-9290-50844e8d5561\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") "
Nov 03 21:16:14.129588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.129555    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c\" (UniqueName: \"kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c\") pod \"21904997-954f-4cba-9290-50844e8d5561\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") "
Nov 03 21:16:14.129665 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.129621    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca\") pod \"21904997-954f-4cba-9290-50844e8d5561\" (UID: \"21904997-954f-4cba-9290-50844e8d5561\") "
Nov 03 21:16:14.129978 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.129957    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca" pod "21904997-954f-4cba-9290-50844e8d5561" (UID: "21904997-954f-4cba-9290-50844e8d5561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:14.131535 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.131514    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c" pod "21904997-954f-4cba-9290-50844e8d5561" (UID: "21904997-954f-4cba-9290-50844e8d5561"). InnerVolumeSpecName "kube-api-access-vw65c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:14.131535 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.131523    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca" pod "21904997-954f-4cba-9290-50844e8d5561" (UID: "21904997-954f-4cba-9290-50844e8d5561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:14.149192 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.149171    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:14.230973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.230912    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw65c\" (UniqueName: \"kubernetes.io/projected/21904997-954f-4cba-9290-50844e8d5561-kube-api-access-vw65c\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:14.230973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.230937    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21904997-954f-4cba-9290-50844e8d5561-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:14.230973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.230947    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21904997-954f-4cba-9290-50844e8d5561-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:14.569707 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.569624    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"]
Nov 03 21:16:14.570214 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.570194    5318 cpu_manager.go:414] "containerMap: removing container" podUID="21904997-954f-4cba-9290-50844e8d5561" containerName="revoke-keys"
Nov 03 21:16:14.570320 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.570216    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21904997-954f-4cba-9290-50844e8d5561" containerName="revoke-keys"
Nov 03 21:16:14.570320 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.570271    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main"
Nov 03 21:16:14.570320 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.570287    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main"
Nov 03 21:16:14.570512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.570427    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc35bbb-790d-42f9-9ffb-623c5761b34a" containerName="main"
Nov 03 21:16:14.570512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.570451    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="21904997-954f-4cba-9290-50844e8d5561" containerName="revoke-keys"
Nov 03 21:16:14.574102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.574081    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.633775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.633744    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"]
Nov 03 21:16:14.634243 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.634228    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.634295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.634259    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbkw\" (UniqueName: \"kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.634410 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.634392    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.735158 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.735124    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.735328 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.735256    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.735328 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.735297    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbkw\" (UniqueName: \"kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.735952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.735924    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.737542 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.737517    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.792426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.792400    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbkw\" (UniqueName: \"kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw\") pod \"maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.891030 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.891003    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:14.901531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.901479    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"
Nov 03 21:16:14.901531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.901481    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z" event={"ID":"21904997-954f-4cba-9290-50844e8d5561","Type":"ContainerDied","Data":"159f5d128ab46728be442ba31d114677b9b5e47bbc74e534fea84914fb2eab95"}
Nov 03 21:16:14.901628 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:14.901536    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159f5d128ab46728be442ba31d114677b9b5e47bbc74e534fea84914fb2eab95"
Nov 03 21:16:15.261095 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:15.261043    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"]
Nov 03 21:16:15.908055 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:15.908015    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl" event={"ID":"9d685ab1-56dd-4bae-bede-48da0065c51a","Type":"ContainerStarted","Data":"160ef47e7f2145955d109f9535af4edde04de77287ba8fb39b845dd25919a093"}
Nov 03 21:16:15.908205 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:15.908077    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl" event={"ID":"9d685ab1-56dd-4bae-bede-48da0065c51a","Type":"ContainerStarted","Data":"ad613c058328fe8dbb291bf15ca22a23a1b99f32ef42f0970bbdd9bae97d2256"}
Nov 03 21:16:16.050162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:16.050096    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl" podStartSLOduration=2.050084051 podStartE2EDuration="2.050084051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:16.049703134 +0000 UTC m=+2769.029259973" watchObservedRunningTime="2026-11-03 21:16:16.050084051 +0000 UTC m=+2769.029640891"
Nov 03 21:16:16.156199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:16.156171    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:16.915477 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:16.915447    5318 generic.go:356] "Generic (PLEG): container finished" podID="9d685ab1-56dd-4bae-bede-48da0065c51a" containerID="160ef47e7f2145955d109f9535af4edde04de77287ba8fb39b845dd25919a093" exitCode=0
Nov 03 21:16:16.915856 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:16.915531    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl" event={"ID":"9d685ab1-56dd-4bae-bede-48da0065c51a","Type":"ContainerDied","Data":"160ef47e7f2145955d109f9535af4edde04de77287ba8fb39b845dd25919a093"}
Nov 03 21:16:17.831821 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:17.831784    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:18.053356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.052424    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:18.080795 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.080769    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:18.170254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.170211    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca\") pod \"9d685ab1-56dd-4bae-bede-48da0065c51a\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") "
Nov 03 21:16:18.170761 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.170734    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca\") pod \"9d685ab1-56dd-4bae-bede-48da0065c51a\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") "
Nov 03 21:16:18.170927 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.170786    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw\" (UniqueName: \"kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw\") pod \"9d685ab1-56dd-4bae-bede-48da0065c51a\" (UID: \"9d685ab1-56dd-4bae-bede-48da0065c51a\") "
Nov 03 21:16:18.171379 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.171303    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca" pod "9d685ab1-56dd-4bae-bede-48da0065c51a" (UID: "9d685ab1-56dd-4bae-bede-48da0065c51a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:18.173991 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.173969    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw" pod "9d685ab1-56dd-4bae-bede-48da0065c51a" (UID: "9d685ab1-56dd-4bae-bede-48da0065c51a"). InnerVolumeSpecName "kube-api-access-cqbkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:18.174366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.174305    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca" pod "9d685ab1-56dd-4bae-bede-48da0065c51a" (UID: "9d685ab1-56dd-4bae-bede-48da0065c51a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:18.252220 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.252195    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:18.272673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.272647    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:18.272673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.272671    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d685ab1-56dd-4bae-bede-48da0065c51a-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:18.272817 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.272680    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqbkw\" (UniqueName: \"kubernetes.io/projected/9d685ab1-56dd-4bae-bede-48da0065c51a-kube-api-access-cqbkw\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:18.836551 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.836521    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:18.928037 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.928008    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl" event={"ID":"9d685ab1-56dd-4bae-bede-48da0065c51a","Type":"ContainerDied","Data":"ad613c058328fe8dbb291bf15ca22a23a1b99f32ef42f0970bbdd9bae97d2256"}
Nov 03 21:16:18.928225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.928061    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad613c058328fe8dbb291bf15ca22a23a1b99f32ef42f0970bbdd9bae97d2256"
Nov 03 21:16:18.928225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:18.928015    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"
Nov 03 21:16:19.006507 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:19.006478    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:19.058013 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:19.057982    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"]
Nov 03 21:16:19.207428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:19.207315    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:20.602366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:20.602325    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:23.223440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:23.223404    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:25.171471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:25.171437    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:25.379110 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:25.379043    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:25.586159 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:25.586072    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:25.934391 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:25.934331    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:16:28.318640 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.318602    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:28.798255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.798224    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:16:28.885690 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.885664    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69\" (UniqueName: \"kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69\") pod \"3eb9d38e-07f3-4993-82ee-732d30c16095\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") "
Nov 03 21:16:28.885855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.885818    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume\") pod \"3eb9d38e-07f3-4993-82ee-732d30c16095\" (UID: \"3eb9d38e-07f3-4993-82ee-732d30c16095\") "
Nov 03 21:16:28.886261 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.886235    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume" pod "3eb9d38e-07f3-4993-82ee-732d30c16095" (UID: "3eb9d38e-07f3-4993-82ee-732d30c16095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:28.887648 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.887625    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69" pod "3eb9d38e-07f3-4993-82ee-732d30c16095" (UID: "3eb9d38e-07f3-4993-82ee-732d30c16095"). InnerVolumeSpecName "kube-api-access-bbx69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:28.889177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.889152    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:28.987091 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.987001    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb9d38e-07f3-4993-82ee-732d30c16095-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:28.987091 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.987038    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbx69\" (UniqueName: \"kubernetes.io/projected/3eb9d38e-07f3-4993-82ee-732d30c16095-kube-api-access-bbx69\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:28.987903 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.987874    5318 generic.go:356] "Generic (PLEG): container finished" podID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerID="3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e" exitCode=0
Nov 03 21:16:28.987997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.987924    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"
Nov 03 21:16:28.987997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.987960    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" event={"ID":"3eb9d38e-07f3-4993-82ee-732d30c16095","Type":"ContainerDied","Data":"3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e"}
Nov 03 21:16:28.988077 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.988018    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458" event={"ID":"3eb9d38e-07f3-4993-82ee-732d30c16095","Type":"ContainerDied","Data":"af7c9b4015a7fd0351b4504e64931629dd55e454ecb24977afdaf4f39538e672"}
Nov 03 21:16:28.988077 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.988047    5318 scope.go:122] "RemoveContainer" containerID="3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e"
Nov 03 21:16:28.999375 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.999356    5318 scope.go:122] "RemoveContainer" containerID="3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e"
Nov 03 21:16:28.999679 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:28.999651    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e\": container with ID starting with 3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e not found: ID does not exist" containerID="3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e"
Nov 03 21:16:28.999772 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:28.999689    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e"} err="failed to get container status \"3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e\": rpc error: code = NotFound desc = could not find container \"3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e\": container with ID starting with 3c8e6174168cd4a69f577821fdcf559d1e865c7469d30ddc19ace2b7862a672e not found: ID does not exist"
Nov 03 21:16:29.141248 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:29.141212    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"]
Nov 03 21:16:29.146370 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:29.146331    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-df71d7-5ccd8b5495-87458"]
Nov 03 21:16:29.257069 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:29.256988    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" path="/var/lib/kubelet/pods/3eb9d38e-07f3-4993-82ee-732d30c16095/volumes"
Nov 03 21:16:30.450073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:30.450042    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:16:33.432596 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:33.432559    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:33.831234 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:33.831146    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:34.038604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.038567    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:34.082797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.082691    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main" containerID="cri-o://c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e" gracePeriod=45
Nov 03 21:16:34.361026 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.360994    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 21:16:34.445144 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445110    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache\") pod \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") "
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445155    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff\" (UniqueName: \"kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff\") pod \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") "
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445215    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir\") pod \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") "
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445233    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm\") pod \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") "
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445258    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home\") pod \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") "
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445368    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs\") pod \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\" (UID: \"cf7441cb-533b-4e3c-90f5-1853bb3e090b\") "
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445503    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache" pod "cf7441cb-533b-4e3c-90f5-1853bb3e090b" (UID: "cf7441cb-533b-4e3c-90f5-1853bb3e090b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445544    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home" pod "cf7441cb-533b-4e3c-90f5-1853bb3e090b" (UID: "cf7441cb-533b-4e3c-90f5-1853bb3e090b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.445641 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445555    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir" pod "cf7441cb-533b-4e3c-90f5-1853bb3e090b" (UID: "cf7441cb-533b-4e3c-90f5-1853bb3e090b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.446047 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445884    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.446047 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445907    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.446047 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.445923    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.447643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.447614    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm" pod "cf7441cb-533b-4e3c-90f5-1853bb3e090b" (UID: "cf7441cb-533b-4e3c-90f5-1853bb3e090b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:16:34.447833 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.447814    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs" pod "cf7441cb-533b-4e3c-90f5-1853bb3e090b" (UID: "cf7441cb-533b-4e3c-90f5-1853bb3e090b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:34.447914 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.447817    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff" pod "cf7441cb-533b-4e3c-90f5-1853bb3e090b" (UID: "cf7441cb-533b-4e3c-90f5-1853bb3e090b"). InnerVolumeSpecName "kube-api-access-k2vff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:34.547107 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.547069    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vff\" (UniqueName: \"kubernetes.io/projected/cf7441cb-533b-4e3c-90f5-1853bb3e090b-kube-api-access-k2vff\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.547107 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.547111    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7441cb-533b-4e3c-90f5-1853bb3e090b-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:34.547293 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:34.547127    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7441cb-533b-4e3c-90f5-1853bb3e090b-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:35.028883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.028849    5318 generic.go:356] "Generic (PLEG): container finished" podID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerID="c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e" exitCode=0
Nov 03 21:16:35.029067 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.028917    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"
Nov 03 21:16:35.029067 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.028948    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" event={"ID":"cf7441cb-533b-4e3c-90f5-1853bb3e090b","Type":"ContainerDied","Data":"c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e"}
Nov 03 21:16:35.029067 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.029005    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29" event={"ID":"cf7441cb-533b-4e3c-90f5-1853bb3e090b","Type":"ContainerDied","Data":"d4f9ec6591eebf575774890276acac434026e5d163844d6fb5cf6a4c6472f0c3"}
Nov 03 21:16:35.029067 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.029029    5318 scope.go:122] "RemoveContainer" containerID="c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e"
Nov 03 21:16:35.038821 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.038802    5318 scope.go:122] "RemoveContainer" containerID="c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e"
Nov 03 21:16:35.039118 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:35.039095    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e\": container with ID starting with c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e not found: ID does not exist" containerID="c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e"
Nov 03 21:16:35.039227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.039123    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e"} err="failed to get container status \"c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e\": rpc error: code = NotFound desc = could not find container \"c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e\": container with ID starting with c635656ebc66d9f53c3349833237729eddf7cac373df3c77b49a78f4e4ba333e not found: ID does not exist"
Nov 03 21:16:35.081524 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.081488    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:35.181595 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.181558    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"]
Nov 03 21:16:35.228162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.228127    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/premium-w3-88228a-kserve-7f7bc87958-dkq29"]
Nov 03 21:16:35.254556 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:35.254522    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" path="/var/lib/kubelet/pods/cf7441cb-533b-4e3c-90f5-1853bb3e090b/volumes"
Nov 03 21:16:37.777349 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.777301    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"]
Nov 03 21:16:37.778028 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778000    5318 cpu_manager.go:414] "containerMap: removing container" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main"
Nov 03 21:16:37.778028 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778026    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main"
Nov 03 21:16:37.778241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778057    5318 cpu_manager.go:414] "containerMap: removing container" podUID="9d685ab1-56dd-4bae-bede-48da0065c51a" containerName="revoke-keys"
Nov 03 21:16:37.778241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778064    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d685ab1-56dd-4bae-bede-48da0065c51a" containerName="revoke-keys"
Nov 03 21:16:37.778241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778096    5318 cpu_manager.go:414] "containerMap: removing container" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerName="payload-pre-processing"
Nov 03 21:16:37.778241 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778105    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerName="payload-pre-processing"
Nov 03 21:16:37.778483 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778246    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d685ab1-56dd-4bae-bede-48da0065c51a" containerName="revoke-keys"
Nov 03 21:16:37.778483 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778275    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb9d38e-07f3-4993-82ee-732d30c16095" containerName="payload-pre-processing"
Nov 03 21:16:37.778595 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.778571    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7441cb-533b-4e3c-90f5-1853bb3e090b" containerName="main"
Nov 03 21:16:37.783683 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.783662    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.837430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.837403    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"]
Nov 03 21:16:37.881276 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.881251    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:16:37.881276 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.881252    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:16:37.882703 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.882669    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.882828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.882815    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnq7j\" (UniqueName: \"kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.882910 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.882884    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.884900 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.884881    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"]
Nov 03 21:16:37.976990 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.976957    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"]
Nov 03 21:16:37.983894 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.983863    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.984053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.984030    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq7j\" (UniqueName: \"kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.984129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.984075    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.984548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.984528    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:37.986881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:37.986860    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:38.041250 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:38.041189    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnq7j\" (UniqueName: \"kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j\") pod \"maas-api-revoke-sub-517cab7e-nh5gm\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:38.100479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:38.100442    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:38.311746 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:38.311545    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:38.470059 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:38.470036    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"]
Nov 03 21:16:38.472193 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:16:38.472162    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c4d959_e409_4895_8aa1_4e6b4ba70f7b.slice/crio-dbb24c075fb1173b03cd494e94786a9c59b4c916d5baa4f02cc1855004f5165d WatchSource:0}: Error finding container dbb24c075fb1173b03cd494e94786a9c59b4c916d5baa4f02cc1855004f5165d: Status 404 returned error can't find the container with id dbb24c075fb1173b03cd494e94786a9c59b4c916d5baa4f02cc1855004f5165d
Nov 03 21:16:38.514927 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:38.514901    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:38.712386 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:38.712330    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:39.055920 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:39.055845    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm" event={"ID":"39c4d959-e409-4895-8aa1-4e6b4ba70f7b","Type":"ContainerStarted","Data":"220ef53dbd39d0ad7474da907ee299af214ed73d0fa95fc91e506aea1897353d"}
Nov 03 21:16:39.055920 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:39.055905    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm" event={"ID":"39c4d959-e409-4895-8aa1-4e6b4ba70f7b","Type":"ContainerStarted","Data":"dbb24c075fb1173b03cd494e94786a9c59b4c916d5baa4f02cc1855004f5165d"}
Nov 03 21:16:39.164290 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:39.164201    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm" podStartSLOduration=2.164187993 podStartE2EDuration="2.164187993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:39.163704641 +0000 UTC m=+2792.143261486" watchObservedRunningTime="2026-11-03 21:16:39.164187993 +0000 UTC m=+2792.143744833"
Nov 03 21:16:40.067208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:40.067174    5318 generic.go:356] "Generic (PLEG): container finished" podID="39c4d959-e409-4895-8aa1-4e6b4ba70f7b" containerID="220ef53dbd39d0ad7474da907ee299af214ed73d0fa95fc91e506aea1897353d" exitCode=0
Nov 03 21:16:40.067705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:40.067235    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm" event={"ID":"39c4d959-e409-4895-8aa1-4e6b4ba70f7b","Type":"ContainerDied","Data":"220ef53dbd39d0ad7474da907ee299af214ed73d0fa95fc91e506aea1897353d"}
Nov 03 21:16:41.160992 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.160958    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:41.219885 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.219864    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:41.319169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.319145    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca\") pod \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") "
Nov 03 21:16:41.319315 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.319188    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca\") pod \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") "
Nov 03 21:16:41.319315 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.319255    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j\" (UniqueName: \"kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j\") pod \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\" (UID: \"39c4d959-e409-4895-8aa1-4e6b4ba70f7b\") "
Nov 03 21:16:41.319535 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.319512    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca" pod "39c4d959-e409-4895-8aa1-4e6b4ba70f7b" (UID: "39c4d959-e409-4895-8aa1-4e6b4ba70f7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:41.321311 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.321290    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca" pod "39c4d959-e409-4895-8aa1-4e6b4ba70f7b" (UID: "39c4d959-e409-4895-8aa1-4e6b4ba70f7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:16:41.321444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.321387    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j" pod "39c4d959-e409-4895-8aa1-4e6b4ba70f7b" (UID: "39c4d959-e409-4895-8aa1-4e6b4ba70f7b"). InnerVolumeSpecName "kube-api-access-gnq7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:41.420496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.420468    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:41.420496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.420497    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:41.420724 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:41.420511    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnq7j\" (UniqueName: \"kubernetes.io/projected/39c4d959-e409-4895-8aa1-4e6b4ba70f7b-kube-api-access-gnq7j\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:42.079925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:42.079889    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"
Nov 03 21:16:42.079925 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:42.079919    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm" event={"ID":"39c4d959-e409-4895-8aa1-4e6b4ba70f7b","Type":"ContainerDied","Data":"dbb24c075fb1173b03cd494e94786a9c59b4c916d5baa4f02cc1855004f5165d"}
Nov 03 21:16:42.080137 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:42.079969    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbb24c075fb1173b03cd494e94786a9c59b4c916d5baa4f02cc1855004f5165d"
Nov 03 21:16:42.885750 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:42.885713    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9" podUID="739c317f-11d8-4a9b-bfa9-053dcca3e36c" containerName="payload-processing" containerID="cri-o://8d7f24521a7c503206f86fe86455d9fa997317ddb3b37a344a0b6eb9116ba288" gracePeriod=25
Nov 03 21:16:42.982318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:42.982221    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerName="payload-pre-processing" containerID="cri-o://06b7ad407ca8ec58bc747c9b90e3e459638e3f793907cb4389888c62579eea9d" gracePeriod=25
Nov 03 21:16:42.993745 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:42.993713    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:43.100006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.099975    5318 generic.go:356] "Generic (PLEG): container finished" podID="739c317f-11d8-4a9b-bfa9-053dcca3e36c" containerID="8d7f24521a7c503206f86fe86455d9fa997317ddb3b37a344a0b6eb9116ba288" exitCode=0
Nov 03 21:16:43.100166 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.100091    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9" event={"ID":"739c317f-11d8-4a9b-bfa9-053dcca3e36c","Type":"ContainerDied","Data":"8d7f24521a7c503206f86fe86455d9fa997317ddb3b37a344a0b6eb9116ba288"}
Nov 03 21:16:43.166391 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.166365    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:16:43.341235 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.341202    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs\" (UniqueName: \"kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs\") pod \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") "
Nov 03 21:16:43.341428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.341267    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume\") pod \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\" (UID: \"739c317f-11d8-4a9b-bfa9-053dcca3e36c\") "
Nov 03 21:16:43.341710 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.341685    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume" pod "739c317f-11d8-4a9b-bfa9-053dcca3e36c" (UID: "739c317f-11d8-4a9b-bfa9-053dcca3e36c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:43.343401 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.343370    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs" pod "739c317f-11d8-4a9b-bfa9-053dcca3e36c" (UID: "739c317f-11d8-4a9b-bfa9-053dcca3e36c"). InnerVolumeSpecName "kube-api-access-4wwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:43.442501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.442421    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739c317f-11d8-4a9b-bfa9-053dcca3e36c-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:43.442501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:43.442462    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwvs\" (UniqueName: \"kubernetes.io/projected/739c317f-11d8-4a9b-bfa9-053dcca3e36c-kube-api-access-4wwvs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:44.107503 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.107470    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9" event={"ID":"739c317f-11d8-4a9b-bfa9-053dcca3e36c","Type":"ContainerDied","Data":"41b1aa9cd4ce7eda0be822cc67dab691888878e45e3dd8921dcd5fedf3b35a17"}
Nov 03 21:16:44.107503 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.107486    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"
Nov 03 21:16:44.107952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.107557    5318 scope.go:122] "RemoveContainer" containerID="8d7f24521a7c503206f86fe86455d9fa997317ddb3b37a344a0b6eb9116ba288"
Nov 03 21:16:44.261575 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.261527    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"]
Nov 03 21:16:44.307721 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.307685    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fc4e8d-6468895887-bjsx9"]
Nov 03 21:16:44.363901 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.363822    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:44.577081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.577046    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:44.812924 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:44.812848    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:45.018827 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.018793    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:45.048605 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.048574    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"]
Nov 03 21:16:45.093268 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.093243    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3b8feb-94kj8"]
Nov 03 21:16:45.255803 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.255759    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be203ef-5dd1-43b6-9430-9bf364b73d84" path="/var/lib/kubelet/pods/5be203ef-5dd1-43b6-9430-9bf364b73d84/volumes"
Nov 03 21:16:45.256179 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.256160    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739c317f-11d8-4a9b-bfa9-053dcca3e36c" path="/var/lib/kubelet/pods/739c317f-11d8-4a9b-bfa9-053dcca3e36c/volumes"
Nov 03 21:16:45.326257 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.326231    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:45.533042 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:45.532938    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:47.464534 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:47.464479    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:47.668066 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:47.668033    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:50.308350 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:50.308302    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:51.822615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:51.822579    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:52.029566 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:52.029534    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:54.363511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:54.363478    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:16:55.563941 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:55.563905    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:55.776471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:55.776438    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:55.935617 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:16:55.935584    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:16:56.017179 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:56.017144    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:16:57.417875 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:16:57.417842    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:00.308264 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:00.308229    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:01.892301 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:01.892261    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:02.129936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:02.129906    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:02.343515 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:02.343490    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:02.803739 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:02.803664    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:03.220805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.220767    5318 generic.go:356] "Generic (PLEG): container finished" podID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerID="06b7ad407ca8ec58bc747c9b90e3e459638e3f793907cb4389888c62579eea9d" exitCode=0
Nov 03 21:17:03.221165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.220981    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" event={"ID":"77acde91-fa20-44cc-9b8d-bfa0ff436811","Type":"ContainerDied","Data":"06b7ad407ca8ec58bc747c9b90e3e459638e3f793907cb4389888c62579eea9d"}
Nov 03 21:17:03.252860 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.252837    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:17:03.349136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.349109    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume\") pod \"77acde91-fa20-44cc-9b8d-bfa0ff436811\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") "
Nov 03 21:17:03.349357 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.349145    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq\" (UniqueName: \"kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq\") pod \"77acde91-fa20-44cc-9b8d-bfa0ff436811\" (UID: \"77acde91-fa20-44cc-9b8d-bfa0ff436811\") "
Nov 03 21:17:03.349519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.349494    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume" pod "77acde91-fa20-44cc-9b8d-bfa0ff436811" (UID: "77acde91-fa20-44cc-9b8d-bfa0ff436811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:03.351467 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.351443    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq" pod "77acde91-fa20-44cc-9b8d-bfa0ff436811" (UID: "77acde91-fa20-44cc-9b8d-bfa0ff436811"). InnerVolumeSpecName "kube-api-access-cbmdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:03.451165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.451122    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77acde91-fa20-44cc-9b8d-bfa0ff436811-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:03.451165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.451161    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmdq\" (UniqueName: \"kubernetes.io/projected/77acde91-fa20-44cc-9b8d-bfa0ff436811-kube-api-access-cbmdq\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:03.667323 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:03.667249    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:04.228265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:04.228234    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm" event={"ID":"77acde91-fa20-44cc-9b8d-bfa0ff436811","Type":"ContainerDied","Data":"dd2216d7065b8964513817a68a57079b916f9ac69b379fa135e328e40a5fbac9"}
Nov 03 21:17:04.228265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:04.228250    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"
Nov 03 21:17:04.228780 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:04.228291    5318 scope.go:122] "RemoveContainer" containerID="06b7ad407ca8ec58bc747c9b90e3e459638e3f793907cb4389888c62579eea9d"
Nov 03 21:17:04.381122 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:04.381080    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"]
Nov 03 21:17:04.427802 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:04.427775    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fc4e8d-7d65b7474c-w2pmm"]
Nov 03 21:17:05.254973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:05.254937    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" path="/var/lib/kubelet/pods/77acde91-fa20-44cc-9b8d-bfa0ff436811/volumes"
Nov 03 21:17:07.532761 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:07.532725    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:09.478353 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:09.478307    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:09.711279 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:09.704249    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:09.903943 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:09.903912    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:10.251738 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:10.251645    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:11.309142 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:11.309110    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:17:12.082095 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:12.082062    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:13.767304 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:13.767270    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:13.976907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:13.976868    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:14.183648 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:14.183615    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:14.253048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:14.253016    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"]
Nov 03 21:17:14.299317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:14.299283    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-88228a-lb2nl"]
Nov 03 21:17:14.738253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:14.738219    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:15.255444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:15.255409    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d685ab1-56dd-4bae-bede-48da0065c51a" path="/var/lib/kubelet/pods/9d685ab1-56dd-4bae-bede-48da0065c51a/volumes"
Nov 03 21:17:15.378169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:15.378119    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:19.314429 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:19.314398    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:19.552410 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:19.552378    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:20.884694 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:20.884659    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:21.087533 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:21.087467    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:21.156132 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:21.156050    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"]
Nov 03 21:17:21.156375 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:21.156348    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy" containerID="cri-o://7afbcf1629deb1d439c57d8dc4c402f4c77bb23ffae4efe4d9efde374bfaa2d8" gracePeriod=30
Nov 03 21:17:21.301033 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:21.300973    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:22.716120 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:22.716082    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:22.920620 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:22.920588    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:24.293593 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:24.293562    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:25.933269 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:17:25.933230    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:17:26.355973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.355858    5318 generic.go:356] "Generic (PLEG): container finished" podID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerID="7afbcf1629deb1d439c57d8dc4c402f4c77bb23ffae4efe4d9efde374bfaa2d8" exitCode=0
Nov 03 21:17:26.355973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.355967    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" event={"ID":"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142","Type":"ContainerDied","Data":"7afbcf1629deb1d439c57d8dc4c402f4c77bb23ffae4efe4d9efde374bfaa2d8"}
Nov 03 21:17:26.418854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.418831    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 21:17:26.484668 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484642    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.484783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484672    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.484783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484694    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.484783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484732    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.484929 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484838    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.484929 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484871    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.484929 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484901    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.485088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484928    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.485088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484975    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token\") pod \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\" (UID: \"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142\") "
Nov 03 21:17:26.485197 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484896    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.485265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.484970    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.485265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485103    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.485265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485146    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.485453 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485313    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:26.485514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485446    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.485514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485469    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.485514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485484    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.485514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485497    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.485514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.485510    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.487080 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.487055    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "kube-api-access-9nwpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:26.487190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.487075    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:17:26.487190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.487138    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:26.487190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.487136    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token" pod "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" (UID: "f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:26.586045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.585967    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.586045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.585997    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwpg\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-kube-api-access-9nwpg\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.586045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.586013    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.586045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.586026    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:26.798995 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.798961    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"]
Nov 03 21:17:26.799217 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:26.799194    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" containerID="cri-o://bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b" gracePeriod=30
Nov 03 21:17:27.362719 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:27.362685    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75" event={"ID":"f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142","Type":"ContainerDied","Data":"dd9333da6ee30300accecd96690421a95ba2db9c53f6a1979a24f1d591bcf806"}
Nov 03 21:17:27.363161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:27.362727    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"
Nov 03 21:17:27.363161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:27.362741    5318 scope.go:122] "RemoveContainer" containerID="7afbcf1629deb1d439c57d8dc4c402f4c77bb23ffae4efe4d9efde374bfaa2d8"
Nov 03 21:17:27.516586 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:27.516520    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"]
Nov 03 21:17:27.562416 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:27.562389    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-zpd75"]
Nov 03 21:17:29.218606 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:29.218578    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:29.255353 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:29.255315    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" path="/var/lib/kubelet/pods/f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142/volumes"
Nov 03 21:17:29.561072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:29.560957    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:31.452259 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:31.452225    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:17:31.452666 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:31.452294    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:17:32.067956 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.067898    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:17:32.137467 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137439    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137473    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137493    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137541    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137566    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137596    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137669    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137736    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137788    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket\") pod \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\" (UID: \"fb1e7eff-b179-453a-ac54-1197a1b45f0f\") "
Nov 03 21:17:32.137916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137838    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:32.137916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137904    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:32.138219 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.137966    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:32.138219 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138053    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:32.138219 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138145    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:32.138412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138231    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.138412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138249    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.138412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138271    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.138412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138287    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.138412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.138302    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.140107 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.140079    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:17:32.140230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.140083    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:17:32.140230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.140180    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:32.140378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.140361    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt" pod "fb1e7eff-b179-453a-ac54-1197a1b45f0f" (UID: "fb1e7eff-b179-453a-ac54-1197a1b45f0f"). InnerVolumeSpecName "kube-api-access-ws8dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:32.239847 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.239820    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.239847 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.239849    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8dt\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-kube-api-access-ws8dt\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.240022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.239864    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.240022 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.239876    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb1e7eff-b179-453a-ac54-1197a1b45f0f-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:32.394447 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.394412    5318 generic.go:356] "Generic (PLEG): container finished" podID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerID="bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b" exitCode=0
Nov 03 21:17:32.394632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.394463    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"
Nov 03 21:17:32.394632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.394492    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" event={"ID":"fb1e7eff-b179-453a-ac54-1197a1b45f0f","Type":"ContainerDied","Data":"bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b"}
Nov 03 21:17:32.394632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.394552    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv" event={"ID":"fb1e7eff-b179-453a-ac54-1197a1b45f0f","Type":"ContainerDied","Data":"b1d313713feabb33bd7c70a35e81afec0a5ac62c39247d109abea36000cba420"}
Nov 03 21:17:32.394632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.394576    5318 scope.go:122] "RemoveContainer" containerID="bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b"
Nov 03 21:17:32.404898 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.404884    5318 scope.go:122] "RemoveContainer" containerID="bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b"
Nov 03 21:17:32.405171 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:17:32.405149    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b\": container with ID starting with bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b not found: ID does not exist" containerID="bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b"
Nov 03 21:17:32.405235 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.405177    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b"} err="failed to get container status \"bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b\": rpc error: code = NotFound desc = could not find container \"bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b\": container with ID starting with bb015dc4c127c6ec195b2ab46fa5cce9cf2ae4397f6d03e1d6c9f7118baa603b not found: ID does not exist"
Nov 03 21:17:32.547770 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.547729    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"]
Nov 03 21:17:32.551917 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:32.551892    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kh9nv"]
Nov 03 21:17:33.254349 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:33.254310    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" path="/var/lib/kubelet/pods/fb1e7eff-b179-453a-ac54-1197a1b45f0f/volumes"
Nov 03 21:17:34.526176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:34.526136    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:38.618716 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:38.618683    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:17:39.695723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:39.695693    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:44.631175 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:44.631145    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:49.521893 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:49.521846    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:52.284375 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:52.284315    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:52.490695 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:52.490642    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:53.079664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:53.079627    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:53.699212 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:53.699171    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"]
Nov 03 21:17:53.699729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:53.699441    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" podUID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" containerName="maas-api" containerID="cri-o://8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882" gracePeriod=30
Nov 03 21:17:53.728044 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:53.728012    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" podUID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": dial tcp 10.134.0.42:8443: connect: connection refused"
Nov 03 21:17:53.876796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:53.876760    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:53.977952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:53.977874    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 21:17:54.053459 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.053423    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\") pod \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") "
Nov 03 21:17:54.053625 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.053516    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs\") pod \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") "
Nov 03 21:17:54.053625 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.053549    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt\" (UniqueName: \"kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt\") pod \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") "
Nov 03 21:17:54.053758 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.053666    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca\") pod \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\" (UID: \"d5db2eed-a5b9-4bc0-bc77-0f90911207cd\") "
Nov 03 21:17:54.054006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.053982    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca" pod "d5db2eed-a5b9-4bc0-bc77-0f90911207cd" (UID: "d5db2eed-a5b9-4bc0-bc77-0f90911207cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:17:54.055793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.055770    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls" pod "d5db2eed-a5b9-4bc0-bc77-0f90911207cd" (UID: "d5db2eed-a5b9-4bc0-bc77-0f90911207cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:54.055916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.055839    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt" pod "d5db2eed-a5b9-4bc0-bc77-0f90911207cd" (UID: "d5db2eed-a5b9-4bc0-bc77-0f90911207cd"). InnerVolumeSpecName "kube-api-access-m9bpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:17:54.055916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.055891    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs" pod "d5db2eed-a5b9-4bc0-bc77-0f90911207cd" (UID: "d5db2eed-a5b9-4bc0-bc77-0f90911207cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:17:54.155161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.155130    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.155161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.155156    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.155161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.155165    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.155417 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.155176    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bpt\" (UniqueName: \"kubernetes.io/projected/d5db2eed-a5b9-4bc0-bc77-0f90911207cd-kube-api-access-m9bpt\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:17:54.522384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.522285    5318 generic.go:356] "Generic (PLEG): container finished" podID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" containerID="8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882" exitCode=0
Nov 03 21:17:54.522384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.522355    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"
Nov 03 21:17:54.522384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.522370    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" event={"ID":"d5db2eed-a5b9-4bc0-bc77-0f90911207cd","Type":"ContainerDied","Data":"8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882"}
Nov 03 21:17:54.522674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.522431    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b" event={"ID":"d5db2eed-a5b9-4bc0-bc77-0f90911207cd","Type":"ContainerDied","Data":"5032a825bb0e597e61a4e8ec0da97c32ff1026ca22b1c94333a69f8cb41134c9"}
Nov 03 21:17:54.522674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.522455    5318 scope.go:122] "RemoveContainer" containerID="8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882"
Nov 03 21:17:54.533094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.533073    5318 scope.go:122] "RemoveContainer" containerID="8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882"
Nov 03 21:17:54.533379 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:17:54.533360    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882\": container with ID starting with 8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882 not found: ID does not exist" containerID="8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882"
Nov 03 21:17:54.533434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.533388    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882"} err="failed to get container status \"8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882\": rpc error: code = NotFound desc = could not find container \"8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882\": container with ID starting with 8443d5c393a253b71f082936037d39b7e252b80fbfa889c739b23530ff19c882 not found: ID does not exist"
Nov 03 21:17:54.675888 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.675859    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"]
Nov 03 21:17:54.721980 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:54.721950    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0cd49b-548f6f9866-k9b2b"]
Nov 03 21:17:55.254265 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:55.254229    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" path="/var/lib/kubelet/pods/d5db2eed-a5b9-4bc0-bc77-0f90911207cd/volumes"
Nov 03 21:17:55.473843 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:55.473818    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:55.785701 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:55.785655    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:55.932723 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:17:55.932690    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:17:56.011070 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:56.011038    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:58.956701 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:58.956665    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:17:59.161087 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:17:59.161048    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:00.790202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:00.790174    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:18:00.913669 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:00.913633    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:05.817666 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:05.817630    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:06.476519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:06.476483    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:10.724191 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:10.724153    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:12.906501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:12.906454    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:13.131049 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:13.131013    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:13.343202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:13.343175    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:15.817168 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:15.817092    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:18.713615 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:18.713580    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:18:20.809805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:20.809774    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:24.446870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:24.446831    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:24.668825 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:24.668792    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:25.728515 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:25.728484    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:25.933921 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:18:25.933887    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:18:29.568140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:29.568097    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:30.615642 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:30.615611    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:35.614863 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:35.614830    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:40.514804 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:40.514768    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:45.405369 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:45.405312    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:50.306763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:50.306724    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:55.297617 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:18:55.297586    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:18:55.932561 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:18:55.932529    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:19:00.211797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:00.211766    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:05.113928 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:05.113896    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:10.013714 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:10.013685    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:15.017732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:15.017701    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:18.699482 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:18.699444    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:19.921558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:19.921526    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:24.805699 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:24.805668    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:25.931886 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:19:25.931854    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:19:29.710118 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:29.710086    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:29.916292 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:29.916264    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:34.709474 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:34.709444    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:39.648837 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:39.648807    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:44.560080 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:44.560044    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:49.779941 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:49.779866    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:54.629627 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:54.629594    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:55.411762 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:55.411730    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"]
Nov 03 21:19:55.489017 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:55.488984    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:19:55.933898 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:19:55.933860    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:19:59.623290 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:19:59.623254    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:00.087180 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:00.087151    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:20:04.595131 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:04.595103    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:07.752692 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:07.752661    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:20:07.760083 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:07.760062    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:20:09.512208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:09.512176    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:10.431219 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.431184    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26" podUID="af9e1a77-0705-4e93-80c7-9d7bc980d7b4" containerName="main" containerID="cri-o://4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a" gracePeriod=45
Nov 03 21:20:10.682372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.682302    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:20:10.778783 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.778751    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache\") pod \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") "
Nov 03 21:20:10.778954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.778790    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm\") pod \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") "
Nov 03 21:20:10.778954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.778832    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs\") pod \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") "
Nov 03 21:20:10.778954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.778871    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir\") pod \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") "
Nov 03 21:20:10.778954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.778946    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home\") pod \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") "
Nov 03 21:20:10.779208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.778985    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv\" (UniqueName: \"kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv\") pod \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\" (UID: \"af9e1a77-0705-4e93-80c7-9d7bc980d7b4\") "
Nov 03 21:20:10.779208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.779060    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache" pod "af9e1a77-0705-4e93-80c7-9d7bc980d7b4" (UID: "af9e1a77-0705-4e93-80c7-9d7bc980d7b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:10.779329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.779223    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir" pod "af9e1a77-0705-4e93-80c7-9d7bc980d7b4" (UID: "af9e1a77-0705-4e93-80c7-9d7bc980d7b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:10.779456 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.779429    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:10.779456 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.779454    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:10.779704 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.779684    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home" pod "af9e1a77-0705-4e93-80c7-9d7bc980d7b4" (UID: "af9e1a77-0705-4e93-80c7-9d7bc980d7b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:10.781173 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.781148    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv" pod "af9e1a77-0705-4e93-80c7-9d7bc980d7b4" (UID: "af9e1a77-0705-4e93-80c7-9d7bc980d7b4"). InnerVolumeSpecName "kube-api-access-5dhfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:10.781284 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.781173    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs" pod "af9e1a77-0705-4e93-80c7-9d7bc980d7b4" (UID: "af9e1a77-0705-4e93-80c7-9d7bc980d7b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:10.781284 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.781240    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm" pod "af9e1a77-0705-4e93-80c7-9d7bc980d7b4" (UID: "af9e1a77-0705-4e93-80c7-9d7bc980d7b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:10.880149 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.880113    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:10.880149 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.880140    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:10.880149 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.880149    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:10.880149 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:10.880158    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhfv\" (UniqueName: \"kubernetes.io/projected/af9e1a77-0705-4e93-80c7-9d7bc980d7b4-kube-api-access-5dhfv\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:11.254897 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.254864    5318 generic.go:356] "Generic (PLEG): container finished" podID="af9e1a77-0705-4e93-80c7-9d7bc980d7b4" containerID="4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a" exitCode=0
Nov 03 21:20:11.255073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.254972    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"
Nov 03 21:20:11.255073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.255033    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26" event={"ID":"af9e1a77-0705-4e93-80c7-9d7bc980d7b4","Type":"ContainerDied","Data":"4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a"}
Nov 03 21:20:11.255073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.255057    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26" event={"ID":"af9e1a77-0705-4e93-80c7-9d7bc980d7b4","Type":"ContainerDied","Data":"300eaed79ad4cf4b747844175c767658d94060afea57b33228b19252a8be7f05"}
Nov 03 21:20:11.255202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.255075    5318 scope.go:122] "RemoveContainer" containerID="4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a"
Nov 03 21:20:11.268701 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.268679    5318 scope.go:122] "RemoveContainer" containerID="4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a"
Nov 03 21:20:11.269081 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:20:11.269061    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a\": container with ID starting with 4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a not found: ID does not exist" containerID="4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a"
Nov 03 21:20:11.269176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.269094    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a"} err="failed to get container status \"4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a\": rpc error: code = NotFound desc = could not find container \"4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a\": container with ID starting with 4955a79853d71d5dd973e5afd10881bf84f86247371393424a4985025dd5e76a not found: ID does not exist"
Nov 03 21:20:11.408388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.408347    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"]
Nov 03 21:20:11.454531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:11.454503    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f1855/test-model-1f1855-kserve-99cd448d7-sfv26"]
Nov 03 21:20:13.256937 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:13.256899    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9e1a77-0705-4e93-80c7-9d7bc980d7b4" path="/var/lib/kubelet/pods/af9e1a77-0705-4e93-80c7-9d7bc980d7b4/volumes"
Nov 03 21:20:14.606263 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:14.606230    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:17.119072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:17.119043    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:19.893107 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:19.893064    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:21.779547 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:21.779511    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:20:25.207992 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:25.207961    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:25.934131 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:20:25.934093    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:20:29.172260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:29.172226    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"]
Nov 03 21:20:29.281876 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:29.281841    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"]
Nov 03 21:20:29.616936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:29.616906    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:29.828023 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:29.827995    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:30.138222 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:30.138191    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:30.343997 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:30.343969    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:34.173262 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:34.173198    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerName="payload-processing" containerID="cri-o://d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496" gracePeriod=25
Nov 03 21:20:34.282621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:34.282578    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerName="payload-pre-processing" containerID="cri-o://efe5e2e3e0a5f975f6e7f86721309d98206801fe61b6b836d6a27a305651926c" gracePeriod=25
Nov 03 21:20:34.431953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:34.431873    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"]
Nov 03 21:20:34.432108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:34.432084    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" podUID="b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" containerName="maas-api" containerID="cri-o://37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec" gracePeriod=30
Nov 03 21:20:34.601601 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:34.601567    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:35.682665 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.682643    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:20:35.836004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.835937    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g\" (UniqueName: \"kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g\") pod \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") "
Nov 03 21:20:35.836004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.835979    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\") pod \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") "
Nov 03 21:20:35.836200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.836024    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs\") pod \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") "
Nov 03 21:20:35.836200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.836156    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca\") pod \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\" (UID: \"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f\") "
Nov 03 21:20:35.836540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.836515    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca" pod "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" (UID: "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:35.838084 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.838060    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g" pod "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" (UID: "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f"). InnerVolumeSpecName "kube-api-access-l5t2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:35.838183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.838090    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs" pod "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" (UID: "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:35.838225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.838188    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls" pod "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" (UID: "b4f460c5-e0e9-4a59-a472-5ddedfc51e5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:35.937667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.937630    5318 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-openshift-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:35.937667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.937666    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5t2g\" (UniqueName: \"kubernetes.io/projected/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-kube-api-access-l5t2g\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:35.937844 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.937682    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-maas-api-tls\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:35.937844 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:35.937697    5318 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f-metrics-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:36.182287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.182247    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:36.389652 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.389624    5318 generic.go:356] "Generic (PLEG): container finished" podID="b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" containerID="37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec" exitCode=0
Nov 03 21:20:36.389854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.389684    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"
Nov 03 21:20:36.389854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.389707    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" event={"ID":"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f","Type":"ContainerDied","Data":"37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec"}
Nov 03 21:20:36.389854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.389757    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg" event={"ID":"b4f460c5-e0e9-4a59-a472-5ddedfc51e5f","Type":"ContainerDied","Data":"8c45ff86e69d33664a46fa0e06d66e3b8ce3382a370c49dcb3449015fb36fc75"}
Nov 03 21:20:36.389854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.389777    5318 scope.go:122] "RemoveContainer" containerID="37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec"
Nov 03 21:20:36.398998 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.398953    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:36.400170 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.400147    5318 scope.go:122] "RemoveContainer" containerID="37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec"
Nov 03 21:20:36.400459 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:20:36.400440    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec\": container with ID starting with 37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec not found: ID does not exist" containerID="37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec"
Nov 03 21:20:36.400546 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.400468    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec"} err="failed to get container status \"37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec\": rpc error: code = NotFound desc = could not find container \"37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec\": container with ID starting with 37eaeaedfaabaf675dd7fc33b7b76cf429efcb9c1f438558e1358d3b2d9e8cec not found: ID does not exist"
Nov 03 21:20:36.543893 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.543822    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"]
Nov 03 21:20:36.590083 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.590049    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f1855-77dfdf6876-zmgsg"]
Nov 03 21:20:36.603434 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:36.603403    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:37.182656 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:37.182614    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:20:37.183043 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:37.182663    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:20:37.243501 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:37.243449    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:37.263489 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:37.263453    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" path="/var/lib/kubelet/pods/b4f460c5-e0e9-4a59-a472-5ddedfc51e5f/volumes"
Nov 03 21:20:41.617153 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:41.617121    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:41.984982 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:41.984898    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:20:43.484388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:43.484349    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:43.685210 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:43.685159    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:43.780094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:43.780012    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"]
Nov 03 21:20:43.780330 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:43.780306    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy" containerID="cri-o://5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9" gracePeriod=30
Nov 03 21:20:43.926974 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:43.926924    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:44.681634 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.681603    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"]
Nov 03 21:20:44.682040 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682025    5318 cpu_manager.go:414] "containerMap: removing container" podUID="739c317f-11d8-4a9b-bfa9-053dcca3e36c" containerName="payload-processing"
Nov 03 21:20:44.682094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682040    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739c317f-11d8-4a9b-bfa9-053dcca3e36c" containerName="payload-processing"
Nov 03 21:20:44.682094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682057    5318 cpu_manager.go:414] "containerMap: removing container" podUID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" containerName="maas-api"
Nov 03 21:20:44.682094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682066    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" containerName="maas-api"
Nov 03 21:20:44.682094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682090    5318 cpu_manager.go:414] "containerMap: removing container" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy"
Nov 03 21:20:44.682094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682095    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682105    5318 cpu_manager.go:414] "containerMap: removing container" podUID="39c4d959-e409-4895-8aa1-4e6b4ba70f7b" containerName="revoke-keys"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682110    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c4d959-e409-4895-8aa1-4e6b4ba70f7b" containerName="revoke-keys"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682121    5318 cpu_manager.go:414] "containerMap: removing container" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerName="payload-pre-processing"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682125    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerName="payload-pre-processing"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682150    5318 cpu_manager.go:414] "containerMap: removing container" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682155    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682163    5318 cpu_manager.go:414] "containerMap: removing container" podUID="b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" containerName="maas-api"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682168    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" containerName="maas-api"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682175    5318 cpu_manager.go:414] "containerMap: removing container" podUID="af9e1a77-0705-4e93-80c7-9d7bc980d7b4" containerName="main"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682178    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9e1a77-0705-4e93-80c7-9d7bc980d7b4" containerName="main"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682242    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="77acde91-fa20-44cc-9b8d-bfa0ff436811" containerName="payload-pre-processing"
Nov 03 21:20:44.682254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682254    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1e7eff-b179-453a-ac54-1197a1b45f0f" containerName="istio-proxy"
Nov 03 21:20:44.682643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682261    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f460c5-e0e9-4a59-a472-5ddedfc51e5f" containerName="maas-api"
Nov 03 21:20:44.682643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682267    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9e1a77-0705-4e93-80c7-9d7bc980d7b4" containerName="main"
Nov 03 21:20:44.682643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682279    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="739c317f-11d8-4a9b-bfa9-053dcca3e36c" containerName="payload-processing"
Nov 03 21:20:44.682643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682286    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5db2eed-a5b9-4bc0-bc77-0f90911207cd" containerName="maas-api"
Nov 03 21:20:44.682643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682295    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ee868d-2a9f-4ffb-84e9-ff2afa6b6142" containerName="istio-proxy"
Nov 03 21:20:44.682643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.682304    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c4d959-e409-4895-8aa1-4e6b4ba70f7b" containerName="revoke-keys"
Nov 03 21:20:44.687069 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.687048    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.736000 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.735979    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:20:44.781878 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.781853    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:20:44.781973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.781909    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:20:44.781973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.781911    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:20:44.783942 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.783922    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"]
Nov 03 21:20:44.829513 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.829487    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.829612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.829519    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29gpm\" (UniqueName: \"kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.829715 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.829697    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.931004 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.930976    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.931123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.931011    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29gpm\" (UniqueName: \"kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.931123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.931089    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.931633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.931614    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.933267 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.933229    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.988156 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.988136    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29gpm\" (UniqueName: \"kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm\") pod \"maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:44.999657 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:44.999633    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:45.371401 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:45.371377    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"]
Nov 03 21:20:45.374061 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:20:45.374028    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cd874e_f751_49ce_a7ae_2ec56d73d094.slice/crio-111f75c897ce73444c65a39521b2746794bf7c1bc9e6f3965ab5b0b363e4fd47 WatchSource:0}: Error finding container 111f75c897ce73444c65a39521b2746794bf7c1bc9e6f3965ab5b0b363e4fd47: Status 404 returned error can't find the container with id 111f75c897ce73444c65a39521b2746794bf7c1bc9e6f3965ab5b0b363e4fd47
Nov 03 21:20:45.439844 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:45.439816    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd" event={"ID":"d8cd874e-f751-49ce-a7ae-2ec56d73d094","Type":"ContainerStarted","Data":"111f75c897ce73444c65a39521b2746794bf7c1bc9e6f3965ab5b0b363e4fd47"}
Nov 03 21:20:46.446940 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:46.446905    5318 generic.go:356] "Generic (PLEG): container finished" podID="d8cd874e-f751-49ce-a7ae-2ec56d73d094" containerID="f6630186bd573814e987e74b2d395b065639ee22ce5198446ad49ca722c3cd68" exitCode=0
Nov 03 21:20:46.447373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:46.446991    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd" event={"ID":"d8cd874e-f751-49ce-a7ae-2ec56d73d094","Type":"ContainerDied","Data":"f6630186bd573814e987e74b2d395b065639ee22ce5198446ad49ca722c3cd68"}
Nov 03 21:20:46.866258 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:46.866226    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:47.182666 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.182594    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:20:47.182666 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.182597    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:20:47.602805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.602774    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:47.756360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.756263    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca\") pod \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") "
Nov 03 21:20:47.756537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.756369    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm\" (UniqueName: \"kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm\") pod \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") "
Nov 03 21:20:47.756537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.756408    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca\") pod \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\" (UID: \"d8cd874e-f751-49ce-a7ae-2ec56d73d094\") "
Nov 03 21:20:47.756673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.756606    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca" pod "d8cd874e-f751-49ce-a7ae-2ec56d73d094" (UID: "d8cd874e-f751-49ce-a7ae-2ec56d73d094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:47.756673 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.756665    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:47.758473 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.758448    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm" pod "d8cd874e-f751-49ce-a7ae-2ec56d73d094" (UID: "d8cd874e-f751-49ce-a7ae-2ec56d73d094"). InnerVolumeSpecName "kube-api-access-29gpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:47.758562 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.758476    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca" pod "d8cd874e-f751-49ce-a7ae-2ec56d73d094" (UID: "d8cd874e-f751-49ce-a7ae-2ec56d73d094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:20:47.857965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.857941    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29gpm\" (UniqueName: \"kubernetes.io/projected/d8cd874e-f751-49ce-a7ae-2ec56d73d094-kube-api-access-29gpm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:47.857965 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:47.857964    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cd874e-f751-49ce-a7ae-2ec56d73d094-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:48.457855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:48.457815    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"
Nov 03 21:20:48.458061 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:48.457874    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd" event={"ID":"d8cd874e-f751-49ce-a7ae-2ec56d73d094","Type":"ContainerDied","Data":"111f75c897ce73444c65a39521b2746794bf7c1bc9e6f3965ab5b0b363e4fd47"}
Nov 03 21:20:48.458061 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:48.457929    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="111f75c897ce73444c65a39521b2746794bf7c1bc9e6f3965ab5b0b363e4fd47"
Nov 03 21:20:49.033032 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.033010    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:20:49.172430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172405    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172438    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172460    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172493    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172530    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172621    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172653    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172679    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.172737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172678    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.172737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172708    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb\") pod \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\" (UID: \"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b\") "
Nov 03 21:20:49.173003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172854    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.173003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172942    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.173003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.172854    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:49.173175 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.173057    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.173237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.173189    5318 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-data\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.173237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.173208    5318 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istiod-ca-cert\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.173237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.173223    5318 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.173237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.173236    5318 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-credential-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.173446 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.173244    5318 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-workload-socket\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.174612 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.174587    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:20:49.174852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.174832    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:49.175056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.175041    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:20:49.175117 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.175074    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb" pod "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" (UID: "fe3c10cb-8e12-4f51-80f9-ab41f5cb140b"). InnerVolumeSpecName "kube-api-access-54mtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:49.274625 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.274600    5318 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-token\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.274625 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.274623    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mtb\" (UniqueName: \"kubernetes.io/projected/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-kube-api-access-54mtb\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.274765 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.274633    5318 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-podinfo\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.274765 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.274642    5318 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b-istio-envoy\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:49.464439 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.464362    5318 generic.go:356] "Generic (PLEG): container finished" podID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerID="5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9" exitCode=0
Nov 03 21:20:49.464439 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.464416    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"
Nov 03 21:20:49.464622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.464435    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" event={"ID":"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b","Type":"ContainerDied","Data":"5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9"}
Nov 03 21:20:49.464622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.464492    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g" event={"ID":"fe3c10cb-8e12-4f51-80f9-ab41f5cb140b","Type":"ContainerDied","Data":"1b90e54754fd9dfa6587216e1c8f372ec31f010fbb9d4a6dadafe874113fa7c5"}
Nov 03 21:20:49.464622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.464524    5318 scope.go:122] "RemoveContainer" containerID="5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9"
Nov 03 21:20:49.473736 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.473717    5318 scope.go:122] "RemoveContainer" containerID="5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9"
Nov 03 21:20:49.474007 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:20:49.473990    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9\": container with ID starting with 5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9 not found: ID does not exist" containerID="5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9"
Nov 03 21:20:49.474068 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.474013    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9"} err="failed to get container status \"5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9\": rpc error: code = NotFound desc = could not find container \"5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9\": container with ID starting with 5691c80e903b5138da61ab2b7865cb823ad402577db5588f706b7aa1b902a7a9 not found: ID does not exist"
Nov 03 21:20:49.618106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.618079    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"]
Nov 03 21:20:49.665298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:49.665271    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-bfc9g"]
Nov 03 21:20:51.254463 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:51.254427    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" path="/var/lib/kubelet/pods/fe3c10cb-8e12-4f51-80f9-ab41f5cb140b/volumes"
Nov 03 21:20:52.213123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:52.213080    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:54.425003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.424982    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:20:54.501106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.501080    5318 generic.go:356] "Generic (PLEG): container finished" podID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerID="efe5e2e3e0a5f975f6e7f86721309d98206801fe61b6b836d6a27a305651926c" exitCode=0
Nov 03 21:20:54.501252 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.501175    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" event={"ID":"bfa3a3b6-6150-4cb7-9934-d2bee90b9763","Type":"ContainerDied","Data":"efe5e2e3e0a5f975f6e7f86721309d98206801fe61b6b836d6a27a305651926c"}
Nov 03 21:20:54.502565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.502542    5318 generic.go:356] "Generic (PLEG): container finished" podID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerID="d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496" exitCode=0
Nov 03 21:20:54.502724 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.502597    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"
Nov 03 21:20:54.502724 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.502606    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" event={"ID":"e65d8223-fbbd-45fe-a686-8188a262d7d9","Type":"ContainerDied","Data":"d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496"}
Nov 03 21:20:54.502724 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.502645    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl" event={"ID":"e65d8223-fbbd-45fe-a686-8188a262d7d9","Type":"ContainerDied","Data":"b74568d9bc7629de61d8b2e6050f7d2826716f666820c915a520c46912e17cab"}
Nov 03 21:20:54.502724 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.502667    5318 scope.go:122] "RemoveContainer" containerID="d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496"
Nov 03 21:20:54.511209 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.511178    5318 scope.go:122] "RemoveContainer" containerID="d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496"
Nov 03 21:20:54.511613 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:20:54.511591    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496\": container with ID starting with d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496 not found: ID does not exist" containerID="d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496"
Nov 03 21:20:54.511690 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.511621    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496"} err="failed to get container status \"d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496\": rpc error: code = NotFound desc = could not find container \"d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496\": container with ID starting with d075d36865bc4d1b60ba4e4350f0b45986874dc81f8209ec8b8461888f008496 not found: ID does not exist"
Nov 03 21:20:54.522858 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.522835    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml\" (UniqueName: \"kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml\") pod \"e65d8223-fbbd-45fe-a686-8188a262d7d9\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") "
Nov 03 21:20:54.522980 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.522902    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume\") pod \"e65d8223-fbbd-45fe-a686-8188a262d7d9\" (UID: \"e65d8223-fbbd-45fe-a686-8188a262d7d9\") "
Nov 03 21:20:54.523371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.523324    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume" pod "e65d8223-fbbd-45fe-a686-8188a262d7d9" (UID: "e65d8223-fbbd-45fe-a686-8188a262d7d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:54.524727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.524707    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml" pod "e65d8223-fbbd-45fe-a686-8188a262d7d9" (UID: "e65d8223-fbbd-45fe-a686-8188a262d7d9"). InnerVolumeSpecName "kube-api-access-4ptml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:54.531945 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.531925    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:20:54.623509 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.623487    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr\" (UniqueName: \"kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr\") pod \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") "
Nov 03 21:20:54.623639 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.623571    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume\") pod \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\" (UID: \"bfa3a3b6-6150-4cb7-9934-d2bee90b9763\") "
Nov 03 21:20:54.623784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.623769    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ptml\" (UniqueName: \"kubernetes.io/projected/e65d8223-fbbd-45fe-a686-8188a262d7d9-kube-api-access-4ptml\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:54.623833 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.623790    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65d8223-fbbd-45fe-a686-8188a262d7d9-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:54.623931 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.623913    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume" pod "bfa3a3b6-6150-4cb7-9934-d2bee90b9763" (UID: "bfa3a3b6-6150-4cb7-9934-d2bee90b9763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:20:54.625522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.625506    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr" pod "bfa3a3b6-6150-4cb7-9934-d2bee90b9763" (UID: "bfa3a3b6-6150-4cb7-9934-d2bee90b9763"). InnerVolumeSpecName "kube-api-access-mzgtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:54.725114 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.725055    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzgtr\" (UniqueName: \"kubernetes.io/projected/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-kube-api-access-mzgtr\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:54.725114 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.725081    5318 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa3a3b6-6150-4cb7-9934-d2bee90b9763-plugins-config-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:20:54.914850 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.914811    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"]
Nov 03 21:20:54.919123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:54.919095    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f1855-6bdf7f99f7-4pnsl"]
Nov 03 21:20:55.253764 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:55.253733    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" path="/var/lib/kubelet/pods/e65d8223-fbbd-45fe-a686-8188a262d7d9/volumes"
Nov 03 21:20:55.510332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:55.510240    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w" event={"ID":"bfa3a3b6-6150-4cb7-9934-d2bee90b9763","Type":"ContainerDied","Data":"92efe6ac61d8d6525c3202b7c7784f10d4de2942b0a1d6ede267a876df37bfcf"}
Nov 03 21:20:55.510332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:55.510265    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"
Nov 03 21:20:55.510332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:55.510308    5318 scope.go:122] "RemoveContainer" containerID="efe5e2e3e0a5f975f6e7f86721309d98206801fe61b6b836d6a27a305651926c"
Nov 03 21:20:55.663225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:55.663194    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"]
Nov 03 21:20:55.709133 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:55.709107    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f1855-6996b6c999-fzx4w"]
Nov 03 21:20:55.932227 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:20:55.932198    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:20:56.642121 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:56.642093    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:56.847624 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:56.847593    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:57.118525 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:57.118496    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:20:57.253942 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:20:57.253911    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" path="/var/lib/kubelet/pods/bfa3a3b6-6150-4cb7-9934-d2bee90b9763/volumes"
Nov 03 21:21:02.014356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:02.014314    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:06.347767 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:06.347734    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:07.985512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:07.985482    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:08.201507 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:08.201435    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:08.412791 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:08.412763    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:08.439179 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:08.439150    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"]
Nov 03 21:21:08.485317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:08.485284    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c971dc-t85gd"]
Nov 03 21:21:09.253877 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:09.253846    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8cd874e-f751-49ce-a7ae-2ec56d73d094" path="/var/lib/kubelet/pods/d8cd874e-f751-49ce-a7ae-2ec56d73d094/volumes"
Nov 03 21:21:11.990557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:11.990529    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:21:13.623317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:13.623286    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:15.044366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:15.044319    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"]
Nov 03 21:21:15.090443 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:15.090414    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7add005a-8qm2z"]
Nov 03 21:21:15.255237 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:15.255202    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21904997-954f-4cba-9290-50844e8d5561" path="/var/lib/kubelet/pods/21904997-954f-4cba-9290-50844e8d5561/volumes"
Nov 03 21:21:18.594379 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:18.594253    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:18.953212 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:18.953093    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:23.609944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:23.609891    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:25.029491 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:25.029454    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:25.320196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:25.320103    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:25.522881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:25.522824    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:25.865104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:25.865042    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:25.932457 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:21:25.932427    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:21:28.875651 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:28.875609    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:30.618832 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:30.618794    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:30.780531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:30.780498    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"]
Nov 03 21:21:30.826658 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:30.826632    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"]
Nov 03 21:21:30.954476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:30.954408    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:34.113930 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:34.113894    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:38.383757 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.383711    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:38.477708 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.477679    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"]
Nov 03 21:21:38.478081 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478067    5318 cpu_manager.go:414] "containerMap: removing container" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerName="payload-processing"
Nov 03 21:21:38.478130 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478082    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerName="payload-processing"
Nov 03 21:21:38.478130 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478117    5318 cpu_manager.go:414] "containerMap: removing container" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerName="payload-pre-processing"
Nov 03 21:21:38.478130 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478123    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerName="payload-pre-processing"
Nov 03 21:21:38.478227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478136    5318 cpu_manager.go:414] "containerMap: removing container" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy"
Nov 03 21:21:38.478227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478141    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy"
Nov 03 21:21:38.478227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478154    5318 cpu_manager.go:414] "containerMap: removing container" podUID="d8cd874e-f751-49ce-a7ae-2ec56d73d094" containerName="revoke-keys"
Nov 03 21:21:38.478227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478159    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8cd874e-f751-49ce-a7ae-2ec56d73d094" containerName="revoke-keys"
Nov 03 21:21:38.478227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478223    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa3a3b6-6150-4cb7-9934-d2bee90b9763" containerName="payload-pre-processing"
Nov 03 21:21:38.478428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478232    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3c10cb-8e12-4f51-80f9-ab41f5cb140b" containerName="istio-proxy"
Nov 03 21:21:38.478428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478241    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8cd874e-f751-49ce-a7ae-2ec56d73d094" containerName="revoke-keys"
Nov 03 21:21:38.478428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.478247    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65d8223-fbbd-45fe-a686-8188a262d7d9" containerName="payload-processing"
Nov 03 21:21:38.481146 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.481126    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.517676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.517646    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbc9\" (UniqueName: \"kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.517825 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.517682    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.517825 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.517812    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.529905 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.529868    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:21:38.575808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.575785    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:21:38.575922 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.575824    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:21:38.575922 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.575879    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:21:38.579692 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.579658    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"]
Nov 03 21:21:38.593134 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.593096    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:38.618603 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.618577    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbc9\" (UniqueName: \"kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.618732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.618616    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.618732 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.618687    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.619418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.619400    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.621020 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.620998    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.675557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.675507    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbc9\" (UniqueName: \"kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9\") pod \"maas-api-revoke-sub-f120b874-wxtvn\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:38.796372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:38.796326    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:39.160695 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:39.160653    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"]
Nov 03 21:21:39.163219 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:21:39.163178    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132f7284_ad5e_4889_80f4_9fa4b2b0cfc1.slice/crio-6197d4b1c7b194c94c8d3764c26adeeeec8be2bda1fc413d35dd0c2979ea10ea WatchSource:0}: Error finding container 6197d4b1c7b194c94c8d3764c26adeeeec8be2bda1fc413d35dd0c2979ea10ea: Status 404 returned error can't find the container with id 6197d4b1c7b194c94c8d3764c26adeeeec8be2bda1fc413d35dd0c2979ea10ea
Nov 03 21:21:39.291883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:39.291857    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:39.495511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:39.495485    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:39.751251 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:39.751163    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn" event={"ID":"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1","Type":"ContainerStarted","Data":"ddb36aa62bfaff18ba6c1efd0fc637d277210d547e3148a1988bb6ca72cd4277"}
Nov 03 21:21:39.751251 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:39.751221    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn" event={"ID":"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1","Type":"ContainerStarted","Data":"6197d4b1c7b194c94c8d3764c26adeeeec8be2bda1fc413d35dd0c2979ea10ea"}
Nov 03 21:21:39.860255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:39.860205    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn" podStartSLOduration=1.860192482 podStartE2EDuration="1.860192482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:39.860048592 +0000 UTC m=+3092.839605433" watchObservedRunningTime="2026-11-03 21:21:39.860192482 +0000 UTC m=+3092.839749319"
Nov 03 21:21:40.758710 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:40.758679    5318 generic.go:356] "Generic (PLEG): container finished" podID="132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" containerID="ddb36aa62bfaff18ba6c1efd0fc637d277210d547e3148a1988bb6ca72cd4277" exitCode=0
Nov 03 21:21:40.759097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:40.758766    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn" event={"ID":"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1","Type":"ContainerDied","Data":"ddb36aa62bfaff18ba6c1efd0fc637d277210d547e3148a1988bb6ca72cd4277"}
Nov 03 21:21:41.900981 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.900952    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:41.950158 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.950135    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9\" (UniqueName: \"kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9\") pod \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") "
Nov 03 21:21:41.950298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.950188    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca\") pod \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") "
Nov 03 21:21:41.950298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.950227    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca\") pod \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\" (UID: \"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1\") "
Nov 03 21:21:41.950621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.950596    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca" pod "132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" (UID: "132f7284-ad5e-4889-80f4-9fa4b2b0cfc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:41.952314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.952292    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9" pod "132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" (UID: "132f7284-ad5e-4889-80f4-9fa4b2b0cfc1"). InnerVolumeSpecName "kube-api-access-wnbc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:41.952314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:41.952299    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca" pod "132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" (UID: "132f7284-ad5e-4889-80f4-9fa4b2b0cfc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:42.051587 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.051514    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbc9\" (UniqueName: \"kubernetes.io/projected/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-kube-api-access-wnbc9\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:42.051587 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.051540    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:42.051587 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.051551    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:42.057578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.057551    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"]
Nov 03 21:21:42.059032 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.059010    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517cab7e-nh5gm"]
Nov 03 21:21:42.770042 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.770012    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"
Nov 03 21:21:42.770211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.770053    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn" event={"ID":"132f7284-ad5e-4889-80f4-9fa4b2b0cfc1","Type":"ContainerDied","Data":"6197d4b1c7b194c94c8d3764c26adeeeec8be2bda1fc413d35dd0c2979ea10ea"}
Nov 03 21:21:42.770211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:42.770098    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6197d4b1c7b194c94c8d3764c26adeeeec8be2bda1fc413d35dd0c2979ea10ea"
Nov 03 21:21:43.256260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:43.256220    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c4d959-e409-4895-8aa1-4e6b4ba70f7b" path="/var/lib/kubelet/pods/39c4d959-e409-4895-8aa1-4e6b4ba70f7b/volumes"
Nov 03 21:21:45.799957 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:45.799896    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" containerID="cri-o://2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6" gracePeriod=45
Nov 03 21:21:45.845727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:45.845679    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" containerID="cri-o://919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137" gracePeriod=45
Nov 03 21:21:46.000366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.000312    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused"
Nov 03 21:21:46.125449 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.125416    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 21:21:46.128194 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.128177    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 21:21:46.188135 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188113    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r\") pod \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") "
Nov 03 21:21:46.188249 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188150    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home\") pod \"459be02a-5217-424b-bf75-65009ee0e706\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") "
Nov 03 21:21:46.188249 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188182    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454\" (UniqueName: \"kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454\") pod \"459be02a-5217-424b-bf75-65009ee0e706\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") "
Nov 03 21:21:46.188332 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188295    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home\") pod \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") "
Nov 03 21:21:46.188407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188374    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache\") pod \"459be02a-5217-424b-bf75-65009ee0e706\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") "
Nov 03 21:21:46.188462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188416    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home" pod "459be02a-5217-424b-bf75-65009ee0e706" (UID: "459be02a-5217-424b-bf75-65009ee0e706"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.188462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188426    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir\") pod \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") "
Nov 03 21:21:46.188563 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188535    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm\") pod \"459be02a-5217-424b-bf75-65009ee0e706\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") "
Nov 03 21:21:46.188609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188564    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home" pod "0cf6b6eb-dcdf-4f67-8612-ad72762077e0" (UID: "0cf6b6eb-dcdf-4f67-8612-ad72762077e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.188609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188576    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs\" (UniqueName: \"kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs\") pod \"459be02a-5217-424b-bf75-65009ee0e706\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") "
Nov 03 21:21:46.188609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188578    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache" pod "459be02a-5217-424b-bf75-65009ee0e706" (UID: "459be02a-5217-424b-bf75-65009ee0e706"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.188778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188620    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs\") pod \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") "
Nov 03 21:21:46.188778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188635    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir" pod "0cf6b6eb-dcdf-4f67-8612-ad72762077e0" (UID: "0cf6b6eb-dcdf-4f67-8612-ad72762077e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.188778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188650    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache\") pod \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") "
Nov 03 21:21:46.188778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188716    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir\") pod \"459be02a-5217-424b-bf75-65009ee0e706\" (UID: \"459be02a-5217-424b-bf75-65009ee0e706\") "
Nov 03 21:21:46.188778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188757    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm\") pod \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\" (UID: \"0cf6b6eb-dcdf-4f67-8612-ad72762077e0\") "
Nov 03 21:21:46.189041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189011    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir" pod "459be02a-5217-424b-bf75-65009ee0e706" (UID: "459be02a-5217-424b-bf75-65009ee0e706"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.189118 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.188976    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache" pod "0cf6b6eb-dcdf-4f67-8612-ad72762077e0" (UID: "0cf6b6eb-dcdf-4f67-8612-ad72762077e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.189181 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189146    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.189181 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189164    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.189181 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189178    5318 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-home\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.189352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189191    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.189352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189206    5318 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tmp-dir\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.189352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.189219    5318 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-model-cache\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.190547 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.190526    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r" pod "0cf6b6eb-dcdf-4f67-8612-ad72762077e0" (UID: "0cf6b6eb-dcdf-4f67-8612-ad72762077e0"). InnerVolumeSpecName "kube-api-access-2dp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:46.191070 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.191041    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm" pod "459be02a-5217-424b-bf75-65009ee0e706" (UID: "459be02a-5217-424b-bf75-65009ee0e706"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.191176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.191099    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs" pod "0cf6b6eb-dcdf-4f67-8612-ad72762077e0" (UID: "0cf6b6eb-dcdf-4f67-8612-ad72762077e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:46.191176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.191135    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454" pod "459be02a-5217-424b-bf75-65009ee0e706" (UID: "459be02a-5217-424b-bf75-65009ee0e706"). InnerVolumeSpecName "kube-api-access-s5454". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:46.191484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.191412    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm" pod "0cf6b6eb-dcdf-4f67-8612-ad72762077e0" (UID: "0cf6b6eb-dcdf-4f67-8612-ad72762077e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:46.191778 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.191752    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs" pod "459be02a-5217-424b-bf75-65009ee0e706" (UID: "459be02a-5217-424b-bf75-65009ee0e706"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.290433    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5454\" (UniqueName: \"kubernetes.io/projected/459be02a-5217-424b-bf75-65009ee0e706-kube-api-access-s5454\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.290458    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459be02a-5217-424b-bf75-65009ee0e706-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.290462 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.290467    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459be02a-5217-424b-bf75-65009ee0e706-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.290705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.290476    5318 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-tls-certs\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.290705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.290485    5318 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-dshm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.290705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.290496    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dp6r\" (UniqueName: \"kubernetes.io/projected/0cf6b6eb-dcdf-4f67-8612-ad72762077e0-kube-api-access-2dp6r\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:46.791943 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.791909    5318 generic.go:356] "Generic (PLEG): container finished" podID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerID="2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6" exitCode=0
Nov 03 21:21:46.792138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.791953    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"
Nov 03 21:21:46.792138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.792001    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" event={"ID":"0cf6b6eb-dcdf-4f67-8612-ad72762077e0","Type":"ContainerDied","Data":"2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6"}
Nov 03 21:21:46.792138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.792043    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl" event={"ID":"0cf6b6eb-dcdf-4f67-8612-ad72762077e0","Type":"ContainerDied","Data":"46f27991d229da7fdfc20420d56b9a55419ee167d0acb16fefb153d3f71db9aa"}
Nov 03 21:21:46.792138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.792070    5318 scope.go:122] "RemoveContainer" containerID="2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6"
Nov 03 21:21:46.793533 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.793510    5318 generic.go:356] "Generic (PLEG): container finished" podID="459be02a-5217-424b-bf75-65009ee0e706" containerID="919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137" exitCode=0
Nov 03 21:21:46.793651 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.793544    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" event={"ID":"459be02a-5217-424b-bf75-65009ee0e706","Type":"ContainerDied","Data":"919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137"}
Nov 03 21:21:46.793651 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.793553    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"
Nov 03 21:21:46.793651 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.793572    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" event={"ID":"459be02a-5217-424b-bf75-65009ee0e706","Type":"ContainerDied","Data":"7362df59a96eb7732848ab2dda2a1e360c368c16caf8a47815776ce5f58c1134"}
Nov 03 21:21:46.803318 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.803158    5318 scope.go:122] "RemoveContainer" containerID="2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6"
Nov 03 21:21:46.803621 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:21:46.803469    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6\": container with ID starting with 2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6 not found: ID does not exist" containerID="2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6"
Nov 03 21:21:46.803621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.803504    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6"} err="failed to get container status \"2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6\": rpc error: code = NotFound desc = could not find container \"2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6\": container with ID starting with 2d54bbf66b9b53c0994b8a0b0730cf335f9383bff58724d03d0c4800570831a6 not found: ID does not exist"
Nov 03 21:21:46.803621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.803529    5318 scope.go:122] "RemoveContainer" containerID="919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137"
Nov 03 21:21:46.817555 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.817535    5318 scope.go:122] "RemoveContainer" containerID="919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137"
Nov 03 21:21:46.817833 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:21:46.817815    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137\": container with ID starting with 919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137 not found: ID does not exist" containerID="919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137"
Nov 03 21:21:46.817882 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.817841    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137"} err="failed to get container status \"919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137\": rpc error: code = NotFound desc = could not find container \"919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137\": container with ID starting with 919c93cb69db13e96966a41312be7552a82685f004d05f0b92872aa4d9443137 not found: ID does not exist"
Nov 03 21:21:46.944842 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.944816    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"]
Nov 03 21:21:46.948216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:46.948196    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-2-w6-504eb0-kserve-778f777cbd-mz7jl"]
Nov 03 21:21:47.050159 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:47.050099    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"]
Nov 03 21:21:47.052926 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:47.052901    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 21:21:47.055601 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:47.055581    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/distinct-w6-504eb0-kserve-79969b455f-8zhpb"]
Nov 03 21:21:47.254514 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:47.254479    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" path="/var/lib/kubelet/pods/0cf6b6eb-dcdf-4f67-8612-ad72762077e0/volumes"
Nov 03 21:21:47.254893 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:47.254879    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459be02a-5217-424b-bf75-65009ee0e706" path="/var/lib/kubelet/pods/459be02a-5217-424b-bf75-65009ee0e706/volumes"
Nov 03 21:21:53.660785 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:53.660746    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:53.883604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:53.883567    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:21:55.193035 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:55.192998    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:55.473472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:55.473389    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:21:55.932330 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:21:55.932296    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:21:56.127275 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:21:56.127242    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:02.746184 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:02.746121    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:02.957544 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:02.957518    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:03.166056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:03.166023    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:07.048676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:07.048647    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:07.263294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:07.263260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:08.328167 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:08.328136    5318 scope.go:122] "RemoveContainer" containerID="5f99a947831264cfc981cccf770e8aeccb4733db1a5768b2696d5b455d656857"
Nov 03 21:22:08.922591 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:08.922561    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:10.184044 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:10.184012    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:22:12.604578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:12.604526    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:22:12.605208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:12.605170    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:22:14.914329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:14.914289    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:19.579960 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:19.579933    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:19.788596 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:19.788568    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:25.931837 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:22:25.931798    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:22:32.845631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:32.845598    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:35.120791 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:35.120757    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:22:43.416066 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:43.416028    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:45.595005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:22:45.594930    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:22:55.932440 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:22:55.932398    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:23:08.356030 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:08.355996    5318 scope.go:122] "RemoveContainer" containerID="160ef47e7f2145955d109f9535af4edde04de77287ba8fb39b845dd25919a093"
Nov 03 21:23:08.364471 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:08.364413    5318 scope.go:122] "RemoveContainer" containerID="220ef53dbd39d0ad7474da907ee299af214ed73d0fa95fc91e506aea1897353d"
Nov 03 21:23:08.372086 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:08.372069    5318 scope.go:122] "RemoveContainer" containerID="098022850b0cbc2ffb561cb75761e44f761aed9ac1e42004f7e10cb04d3a5c9a"
Nov 03 21:23:15.115518 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:15.115472    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:25.932104 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:23:25.932067    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:23:31.296331 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:31.296298    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:23:32.577099 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:32.577066    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:23:44.693859 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.693830    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"]
Nov 03 21:23:44.694378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694239    5318 cpu_manager.go:414] "containerMap: removing container" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main"
Nov 03 21:23:44.694378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694252    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main"
Nov 03 21:23:44.694378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694291    5318 cpu_manager.go:414] "containerMap: removing container" podUID="132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" containerName="revoke-keys"
Nov 03 21:23:44.694378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694296    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" containerName="revoke-keys"
Nov 03 21:23:44.694378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694308    5318 cpu_manager.go:414] "containerMap: removing container" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main"
Nov 03 21:23:44.694378 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694313    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main"
Nov 03 21:23:44.694655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694385    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf6b6eb-dcdf-4f67-8612-ad72762077e0" containerName="main"
Nov 03 21:23:44.694655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694399    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" containerName="revoke-keys"
Nov 03 21:23:44.694655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.694408    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="459be02a-5217-424b-bf75-65009ee0e706" containerName="main"
Nov 03 21:23:44.697563 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.697536    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825145    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825177 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825180    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825208    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825387 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825347    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825395    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825472 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825450    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825540 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825517    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825577 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825537    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.825577 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.825562    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92g8r\" (UniqueName: \"kubernetes.io/projected/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-kube-api-access-92g8r\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.903371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.903345    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"]
Nov 03 21:23:44.926367 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926317    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926493 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926387    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926493 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926438    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926604 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926563    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926662 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926609    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92g8r\" (UniqueName: \"kubernetes.io/projected/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-kube-api-access-92g8r\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926722 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926669    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926722 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926701    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926834 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926753    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.926834 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.926809    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.927159 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.927134    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.927302 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.927172    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.927302 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.927254    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.927459 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.927311    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.927526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.927515    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.928846 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.928824    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.929060 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.929041    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.984233 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.984177    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:44.984359 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:44.984322    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92g8r\" (UniqueName: \"kubernetes.io/projected/7534fda2-73dd-40a8-b44b-d8dc20f7d6c8-kube-api-access-92g8r\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-cqqww\" (UID: \"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:45.010115 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:45.010096    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:45.428771 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:45.428741    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"]
Nov 03 21:23:45.462327 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:23:45.462300    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7534fda2_73dd_40a8_b44b_d8dc20f7d6c8.slice/crio-d3ccf7714f65f041c50875d70707744e2b3fbbb98f6550327ebf84a5055d7604 WatchSource:0}: Error finding container d3ccf7714f65f041c50875d70707744e2b3fbbb98f6550327ebf84a5055d7604: Status 404 returned error can't find the container with id d3ccf7714f65f041c50875d70707744e2b3fbbb98f6550327ebf84a5055d7604
Nov 03 21:23:45.464280 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:45.464252    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:45.464370 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:45.464312    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:45.464370 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:45.464361    5318 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 21:23:46.402875 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:46.402841    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" event={"ID":"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8","Type":"ContainerStarted","Data":"b14f30b3f3d9ae0a2ef11193b1b3e166ff70af19ce9c4c8d5676f3a45a70aa8c"}
Nov 03 21:23:46.403255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:46.402891    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" event={"ID":"7534fda2-73dd-40a8-b44b-d8dc20f7d6c8","Type":"ContainerStarted","Data":"d3ccf7714f65f041c50875d70707744e2b3fbbb98f6550327ebf84a5055d7604"}
Nov 03 21:23:46.564093 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:46.564038    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podStartSLOduration=2.5640216860000002 podStartE2EDuration="2.564021686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:46.562527818 +0000 UTC m=+3219.542084658" watchObservedRunningTime="2026-11-03 21:23:46.564021686 +0000 UTC m=+3219.543578525"
Nov 03 21:23:47.010817 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:47.010784    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:47.012104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:47.012073    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 21:23:47.012242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:47.012120    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 21:23:48.010932 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:48.010901    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 21:23:48.011407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:48.010955    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 21:23:49.010862 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:49.010820    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 21:23:49.011038 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:49.010880    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 21:23:50.010970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:50.010937    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 21:23:50.011216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:50.010989    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 21:23:51.010935 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:51.010902    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 21:23:51.011109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:51.010952    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 21:23:52.011126 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:52.011093    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 21:23:52.011634 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:52.011150    5318 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 21:23:53.570018 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:53.569974    5318 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:53.570630 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:53.570598    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:53.571658 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:53.571636    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww"
Nov 03 21:23:53.981961 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:23:53.981931    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:23:55.936653 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:23:55.936618    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:24:05.399910 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:24:05.399879    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:24:25.933448 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:24:25.933370    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:24:29.352994 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:24:29.352961    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:24:31.628205 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:24:31.628173    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:24:44.371927 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:24:44.371896    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:24:46.633843 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:24:46.633601    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:24:49.202753 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:24:49.202713    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:24:55.932757 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:24:55.932724    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:25:07.793759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:07.793727    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:25:07.799886 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:07.799865    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:25:18.547674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:18.547638    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:21.569179 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:21.569124    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:25.932319 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:25:25.932284    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:25:31.979482 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:31.979442    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:25:36.710271 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:36.710237    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:25:39.593380 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:39.593318    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:25:39.593908 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:39.593421    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-cqqww" podUID="7534fda2-73dd-40a8-b44b-d8dc20f7d6c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:25:47.520440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:25:47.520362    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:25:55.932922 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:25:55.932874    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:26:02.556634 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:02.556596    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:26:25.932580 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:26:25.932543    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:26:26.377156 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:26.377121    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:26:32.321640 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:32.321606    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:38.278005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:38.277974    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:26:42.050557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:42.050518    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"]
Nov 03 21:26:42.096723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:42.096688    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f120b874-wxtvn"]
Nov 03 21:26:43.253436 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:43.253404    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132f7284-ad5e-4889-80f4-9fa4b2b0cfc1" path="/var/lib/kubelet/pods/132f7284-ad5e-4889-80f4-9fa4b2b0cfc1/volumes"
Nov 03 21:26:47.254503 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:47.254472    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:26:55.932632 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:26:55.932597    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:26:57.872344 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:26:57.872313    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:27:08.423354 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:27:08.423316    5318 scope.go:122] "RemoveContainer" containerID="f6630186bd573814e987e74b2d395b065639ee22ce5198446ad49ca722c3cd68"
Nov 03 21:27:10.884123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:27:10.884093    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:27:25.932493 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:27:25.932416    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:27:32.618453 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:27:32.618421    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:27:43.576474 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:27:43.576436    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:27:55.932941 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:27:55.932907    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:28:05.574694 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:28:05.574659    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:28:08.443253 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:28:08.443217    5318 scope.go:122] "RemoveContainer" containerID="ddb36aa62bfaff18ba6c1efd0fc637d277210d547e3148a1988bb6ca72cd4277"
Nov 03 21:28:18.787934 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:28:18.787898    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:28:25.932321 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:28:25.932282    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:28:29.782932 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:28:29.782902    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:28:32.954455 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:28:32.954419    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:28:52.855696 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:28:52.855663    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:28:55.933197 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:28:55.933159    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:29:03.278694 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:29:03.278651    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:29:19.178574 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:29:19.178540    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:29:25.934023 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:29:25.933982    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:29:33.286503 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:29:33.286470    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:29:38.641486 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:29:38.641458    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:29:42.274752 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:29:42.274721    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:29:53.078808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:29:53.078772    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:29:55.933315 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:29:55.933275    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:30:00.158287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.158248    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"]
Nov 03 21:30:00.161956 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.161932    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.210715 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.210681    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:30:00.228180 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.228151    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-b2rmq\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.228362 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.228263    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csm6z\" (UniqueName: \"kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z\") pod \"maas-api-key-cleanup-29895690-b2rmq\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.256797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.256767    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:30:00.256937 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.256844    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:30:00.258550 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.258529    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"]
Nov 03 21:30:00.329563 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.329533    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csm6z\" (UniqueName: \"kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z\") pod \"maas-api-key-cleanup-29895690-b2rmq\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.329726 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.329628    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-b2rmq\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.330521 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.330488    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-b2rmq\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.386955 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.386926    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csm6z\" (UniqueName: \"kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z\") pod \"maas-api-key-cleanup-29895690-b2rmq\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.474763 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.474686    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:00.850345 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:00.850310    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"]
Nov 03 21:30:01.426976 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:01.426889    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" event={"ID":"39e0f0b8-1071-4133-ba73-b0389444d6f4","Type":"ContainerStarted","Data":"f432de63f11adbc6a2d3dd7184c7cffec6da89619e435923a1c2e1fc5e9f295f"}
Nov 03 21:30:01.426976 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:01.426952    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" event={"ID":"39e0f0b8-1071-4133-ba73-b0389444d6f4","Type":"ContainerStarted","Data":"cd016ac782eb0ef0ffa7f3b622e0d8c2b7567a56182253843b711b14d11abd60"}
Nov 03 21:30:02.432798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:02.432764    5318 generic.go:356] "Generic (PLEG): container finished" podID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerID="f432de63f11adbc6a2d3dd7184c7cffec6da89619e435923a1c2e1fc5e9f295f" exitCode=60
Nov 03 21:30:02.433227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:02.432814    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" event={"ID":"39e0f0b8-1071-4133-ba73-b0389444d6f4","Type":"ContainerDied","Data":"f432de63f11adbc6a2d3dd7184c7cffec6da89619e435923a1c2e1fc5e9f295f"}
Nov 03 21:30:02.433227 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:02.433057    5318 scope.go:122] "RemoveContainer" containerID="f432de63f11adbc6a2d3dd7184c7cffec6da89619e435923a1c2e1fc5e9f295f"
Nov 03 21:30:03.439242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:03.439213    5318 generic.go:356] "Generic (PLEG): container finished" podID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerID="43340473e3738ca2dfc20bc258427eea02a1e38ce1a1f10cb76f6668bf832694" exitCode=60
Nov 03 21:30:03.439682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:03.439301    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" event={"ID":"39e0f0b8-1071-4133-ba73-b0389444d6f4","Type":"ContainerDied","Data":"43340473e3738ca2dfc20bc258427eea02a1e38ce1a1f10cb76f6668bf832694"}
Nov 03 21:30:03.439682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:03.439371    5318 scope.go:122] "RemoveContainer" containerID="f432de63f11adbc6a2d3dd7184c7cffec6da89619e435923a1c2e1fc5e9f295f"
Nov 03 21:30:03.439682 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:03.439574    5318 scope.go:122] "RemoveContainer" containerID="43340473e3738ca2dfc20bc258427eea02a1e38ce1a1f10cb76f6668bf832694"
Nov 03 21:30:03.439844 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:30:03.439819    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-b2rmq_odh-ai-gateway-infra(39e0f0b8-1071-4133-ba73-b0389444d6f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4"
Nov 03 21:30:04.445981 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:04.445950    5318 scope.go:122] "RemoveContainer" containerID="43340473e3738ca2dfc20bc258427eea02a1e38ce1a1f10cb76f6668bf832694"
Nov 03 21:30:04.446488 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:30:04.446162    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-b2rmq_odh-ai-gateway-infra(39e0f0b8-1071-4133-ba73-b0389444d6f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4"
Nov 03 21:30:07.827133 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:07.827106    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:30:07.835054 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:07.835034    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:30:13.883884 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:13.883840    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:30:15.248428 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:15.248394    5318 scope.go:122] "RemoveContainer" containerID="43340473e3738ca2dfc20bc258427eea02a1e38ce1a1f10cb76f6668bf832694"
Nov 03 21:30:16.522430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:16.522354    5318 generic.go:356] "Generic (PLEG): container finished" podID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerID="b68e5b9bf2f957647eb1ece5d67858453841bad747579e0e08b5a9c8f1ea7aaf" exitCode=60
Nov 03 21:30:16.522430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:16.522406    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" event={"ID":"39e0f0b8-1071-4133-ba73-b0389444d6f4","Type":"ContainerDied","Data":"b68e5b9bf2f957647eb1ece5d67858453841bad747579e0e08b5a9c8f1ea7aaf"}
Nov 03 21:30:16.522907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:16.522458    5318 scope.go:122] "RemoveContainer" containerID="43340473e3738ca2dfc20bc258427eea02a1e38ce1a1f10cb76f6668bf832694"
Nov 03 21:30:16.522907 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:16.522725    5318 scope.go:122] "RemoveContainer" containerID="b68e5b9bf2f957647eb1ece5d67858453841bad747579e0e08b5a9c8f1ea7aaf"
Nov 03 21:30:16.523007 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:30:16.522950    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-b2rmq_odh-ai-gateway-infra(39e0f0b8-1071-4133-ba73-b0389444d6f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4"
Nov 03 21:30:17.647517 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.647486    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"]
Nov 03 21:30:17.798898 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.798875    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:17.822712 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.822681    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca\") pod \"39e0f0b8-1071-4133-ba73-b0389444d6f4\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") "
Nov 03 21:30:17.822815 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.822782    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z\" (UniqueName: \"kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z\") pod \"39e0f0b8-1071-4133-ba73-b0389444d6f4\" (UID: \"39e0f0b8-1071-4133-ba73-b0389444d6f4\") "
Nov 03 21:30:17.822987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.822967    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca" pod "39e0f0b8-1071-4133-ba73-b0389444d6f4" (UID: "39e0f0b8-1071-4133-ba73-b0389444d6f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:17.823098 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.823080    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0f0b8-1071-4133-ba73-b0389444d6f4-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:17.824679 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.824652    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z" pod "39e0f0b8-1071-4133-ba73-b0389444d6f4" (UID: "39e0f0b8-1071-4133-ba73-b0389444d6f4"). InnerVolumeSpecName "kube-api-access-csm6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:17.923838 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:17.923757    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csm6z\" (UniqueName: \"kubernetes.io/projected/39e0f0b8-1071-4133-ba73-b0389444d6f4-kube-api-access-csm6z\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:30:18.535207 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:18.535182    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"
Nov 03 21:30:18.535366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:18.535179    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq" event={"ID":"39e0f0b8-1071-4133-ba73-b0389444d6f4","Type":"ContainerDied","Data":"cd016ac782eb0ef0ffa7f3b622e0d8c2b7567a56182253843b711b14d11abd60"}
Nov 03 21:30:18.535366 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:18.535317    5318 scope.go:122] "RemoveContainer" containerID="b68e5b9bf2f957647eb1ece5d67858453841bad747579e0e08b5a9c8f1ea7aaf"
Nov 03 21:30:18.687283 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:18.687253    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"]
Nov 03 21:30:18.691448 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:18.691428    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-b2rmq"]
Nov 03 21:30:19.254855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:19.254820    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" path="/var/lib/kubelet/pods/39e0f0b8-1071-4133-ba73-b0389444d6f4/volumes"
Nov 03 21:30:24.889256 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:24.889221    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:30:25.933718 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:30:25.933684    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:30:31.073739 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:31.073704    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:33.172635 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:33.172603    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:33.622852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:33.622824    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:35.713132 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:35.713096    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:35.937400 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:35.937363    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:37.498151 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:37.498089    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:38.525290 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:38.525260    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:39.033705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:39.033673    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:39.237564 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:39.237530    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:42.361714 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:42.361681    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:42.568568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:42.568529    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:42.766518 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:42.766433    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:47.507235 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:47.507201    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:50.560554 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:50.560514    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:30:52.535041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:52.535003    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:30:55.933370 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:30:55.933319    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:30:57.445973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:30:57.445941    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:00.505418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:00.505385    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:01.070633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:01.070597    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:01.277979 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:01.277949    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:02.872002 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:02.871968    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:03.875779 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:03.875741    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:07.803551 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:07.803521    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:08.674087 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:08.674052    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:31:12.816882 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:12.816849    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:17.729048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:17.729016    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:22.710716 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:22.710681    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:25.932912 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:31:25.932878    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:31:27.625531 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:27.625498    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:32.634621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:32.634587    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:33.952190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:33.952156    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:34.298882 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:34.298802    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:31:37.609139 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:37.609107    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:42.610145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:42.610115    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:47.517369 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:47.517281    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:48.381160 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381127    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"]
Nov 03 21:31:48.381528 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381515    5318 cpu_manager.go:414] "containerMap: removing container" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.381576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381529    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.381576 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381570    5318 cpu_manager.go:414] "containerMap: removing container" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.381640 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381589    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.381677 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381650    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.381677 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381660    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.381677 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.381668    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:31:48.385014 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.384995    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.445676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.445652    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:31:48.491863 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.491837    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:31:48.491863 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.491850    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:31:48.492050 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.491922    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:31:48.493588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.493571    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"]
Nov 03 21:31:48.567805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.567778    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzr7\" (UniqueName: \"kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.568140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.567811    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.568140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.567932    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.669086 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.669020    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzr7\" (UniqueName: \"kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.669086 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.669053    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.669260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.669139    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.669812 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.669788    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.671344 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.671320    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.726412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.726388    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzr7\" (UniqueName: \"kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7\") pod \"maas-api-revoke-sub-cc94ac84-vgdkq\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:48.997306 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:48.997226    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:49.362784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:49.362759    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"]
Nov 03 21:31:49.365740 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:31:49.365711    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d719e26_ce33_45e0_9dbd_db135cbf73bb.slice/crio-d8794074b3c0f0ce47d1be7112c8c000ced04dda45390aa825f4b8fe9e86524f WatchSource:0}: Error finding container d8794074b3c0f0ce47d1be7112c8c000ced04dda45390aa825f4b8fe9e86524f: Status 404 returned error can't find the container with id d8794074b3c0f0ce47d1be7112c8c000ced04dda45390aa825f4b8fe9e86524f
Nov 03 21:31:49.988810 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:49.988721    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq" event={"ID":"2d719e26-ce33-45e0-9dbd-db135cbf73bb","Type":"ContainerStarted","Data":"1ab5f88912d3b28998d773743ee3b1e3724638216bef8565976c6b1ba71dc046"}
Nov 03 21:31:49.988810 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:49.988781    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq" event={"ID":"2d719e26-ce33-45e0-9dbd-db135cbf73bb","Type":"ContainerStarted","Data":"d8794074b3c0f0ce47d1be7112c8c000ced04dda45390aa825f4b8fe9e86524f"}
Nov 03 21:31:50.097413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:50.097357    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq" podStartSLOduration=2.097325712 podStartE2EDuration="2.097325712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:50.097002261 +0000 UTC m=+3703.076559101" watchObservedRunningTime="2026-11-03 21:31:50.097325712 +0000 UTC m=+3703.076882551"
Nov 03 21:31:50.994568 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:50.994532    5318 generic.go:356] "Generic (PLEG): container finished" podID="2d719e26-ce33-45e0-9dbd-db135cbf73bb" containerID="1ab5f88912d3b28998d773743ee3b1e3724638216bef8565976c6b1ba71dc046" exitCode=0
Nov 03 21:31:50.994951 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:50.994597    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq" event={"ID":"2d719e26-ce33-45e0-9dbd-db135cbf73bb","Type":"ContainerDied","Data":"1ab5f88912d3b28998d773743ee3b1e3724638216bef8565976c6b1ba71dc046"}
Nov 03 21:31:52.136025 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.136004    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:52.201943 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.201918    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca\") pod \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") "
Nov 03 21:31:52.202048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.201967    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7\" (UniqueName: \"kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7\") pod \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") "
Nov 03 21:31:52.202094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.202073    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca\") pod \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\" (UID: \"2d719e26-ce33-45e0-9dbd-db135cbf73bb\") "
Nov 03 21:31:52.202456 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.202428    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca" pod "2d719e26-ce33-45e0-9dbd-db135cbf73bb" (UID: "2d719e26-ce33-45e0-9dbd-db135cbf73bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:31:52.203896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.203872    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7" pod "2d719e26-ce33-45e0-9dbd-db135cbf73bb" (UID: "2d719e26-ce33-45e0-9dbd-db135cbf73bb"). InnerVolumeSpecName "kube-api-access-2pzr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:31:52.203896 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.203882    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca" pod "2d719e26-ce33-45e0-9dbd-db135cbf73bb" (UID: "2d719e26-ce33-45e0-9dbd-db135cbf73bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:31:52.303397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.303305    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:52.303397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.303353    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzr7\" (UniqueName: \"kubernetes.io/projected/2d719e26-ce33-45e0-9dbd-db135cbf73bb-kube-api-access-2pzr7\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:52.303397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.303363    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d719e26-ce33-45e0-9dbd-db135cbf73bb-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:31:52.520294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:52.520258    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:31:53.008897 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:53.008865    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"
Nov 03 21:31:53.009111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:53.008909    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq" event={"ID":"2d719e26-ce33-45e0-9dbd-db135cbf73bb","Type":"ContainerDied","Data":"d8794074b3c0f0ce47d1be7112c8c000ced04dda45390aa825f4b8fe9e86524f"}
Nov 03 21:31:53.009111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:53.008967    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8794074b3c0f0ce47d1be7112c8c000ced04dda45390aa825f4b8fe9e86524f"
Nov 03 21:31:55.932301 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:31:55.932260    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:31:57.425555 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:31:57.425519    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:00.128442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.128405    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:00.619655 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.619623    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"]
Nov 03 21:32:00.620053 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.620040    5318 cpu_manager.go:414] "containerMap: removing container" podUID="2d719e26-ce33-45e0-9dbd-db135cbf73bb" containerName="revoke-keys"
Nov 03 21:32:00.620102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.620054    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d719e26-ce33-45e0-9dbd-db135cbf73bb" containerName="revoke-keys"
Nov 03 21:32:00.620102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.620075    5318 cpu_manager.go:414] "containerMap: removing container" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:32:00.620102 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.620081    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0f0b8-1071-4133-ba73-b0389444d6f4" containerName="cleanup"
Nov 03 21:32:00.620203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.620156    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d719e26-ce33-45e0-9dbd-db135cbf73bb" containerName="revoke-keys"
Nov 03 21:32:00.624120 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.624092    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.672670 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.672642    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:32:00.676123 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.676100    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbgrm\" (UniqueName: \"kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.676189 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.676163    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.676329 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.676302    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.718808 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.718779    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:32:00.718944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.718812    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:32:00.718944 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.718829    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:32:00.720811 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.720790    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"]
Nov 03 21:32:00.777508 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.777483    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.777621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.777540    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbgrm\" (UniqueName: \"kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.777621 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.777577    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.778138 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.778116    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.779920 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.779899    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.834813 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.834789    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbgrm\" (UniqueName: \"kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:00.937609 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:00.937544    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:01.301393 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:01.301186    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"]
Nov 03 21:32:01.304239 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:32:01.304205    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98e2209_a2ee_430f_9d07_7f92d3e7d1ed.slice/crio-c370c75e6f409de25cd9e1c5df4f280e7727b3d11fdecb1599f1a75ec274cbc1 WatchSource:0}: Error finding container c370c75e6f409de25cd9e1c5df4f280e7727b3d11fdecb1599f1a75ec274cbc1: Status 404 returned error can't find the container with id c370c75e6f409de25cd9e1c5df4f280e7727b3d11fdecb1599f1a75ec274cbc1
Nov 03 21:32:02.055427 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:02.055388    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp" event={"ID":"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed","Type":"ContainerStarted","Data":"cf23de6df7d6723159ddc92704837e0599b34c252751755cf43e1e134b63421d"}
Nov 03 21:32:02.055599 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:02.055454    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp" event={"ID":"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed","Type":"ContainerStarted","Data":"c370c75e6f409de25cd9e1c5df4f280e7727b3d11fdecb1599f1a75ec274cbc1"}
Nov 03 21:32:02.164798 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:02.164745    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp" podStartSLOduration=2.164730861 podStartE2EDuration="2.164730861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:02.164171051 +0000 UTC m=+3715.143727890" watchObservedRunningTime="2026-11-03 21:32:02.164730861 +0000 UTC m=+3715.144287698"
Nov 03 21:32:02.549295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:02.549257    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:03.061953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:03.061925    5318 generic.go:356] "Generic (PLEG): container finished" podID="c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" containerID="cf23de6df7d6723159ddc92704837e0599b34c252751755cf43e1e134b63421d" exitCode=0
Nov 03 21:32:03.062156 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:03.062006    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp" event={"ID":"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed","Type":"ContainerDied","Data":"cf23de6df7d6723159ddc92704837e0599b34c252751755cf43e1e134b63421d"}
Nov 03 21:32:03.376880 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:03.376847    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:32:04.199553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.199529    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:04.311260 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.311225    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca\") pod \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") "
Nov 03 21:32:04.311444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.311367    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca\") pod \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") "
Nov 03 21:32:04.311444 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.311427    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm\" (UniqueName: \"kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm\") pod \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\" (UID: \"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed\") "
Nov 03 21:32:04.311585 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.311559    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca" pod "c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" (UID: "c98e2209-a2ee-430f-9d07-7f92d3e7d1ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:32:04.311774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.311750    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:04.313360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.313311    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca" pod "c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" (UID: "c98e2209-a2ee-430f-9d07-7f92d3e7d1ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:32:04.313457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.313438    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm" pod "c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" (UID: "c98e2209-a2ee-430f-9d07-7f92d3e7d1ed"). InnerVolumeSpecName "kube-api-access-qbgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:32:04.412792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.412747    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:04.412792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:04.412786    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbgrm\" (UniqueName: \"kubernetes.io/projected/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed-kube-api-access-qbgrm\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:32:05.072088 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:05.072057    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:05.072255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:05.072055    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp" event={"ID":"c98e2209-a2ee-430f-9d07-7f92d3e7d1ed","Type":"ContainerDied","Data":"c370c75e6f409de25cd9e1c5df4f280e7727b3d11fdecb1599f1a75ec274cbc1"}
Nov 03 21:32:05.072255 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:05.072198    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c370c75e6f409de25cd9e1c5df4f280e7727b3d11fdecb1599f1a75ec274cbc1"
Nov 03 21:32:05.160165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:05.160132    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:07.818766 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:07.818729    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:12.169228 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:12.169189    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:12.636631 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:12.636581    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:12.839196 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:12.839158    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:16.669319 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:16.669286    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:16.959766 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:16.959681    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:17.242956 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:17.242881    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"]
Nov 03 21:32:17.288747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:17.288709    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"]
Nov 03 21:32:17.298354 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:32:17.298292    5318 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp\" is forbidden: User \"system:node:ip-10-0-135-141.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-141.us-west-2.compute.internal' and this object" podUID="c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-357e69-9vdlp"
Nov 03 21:32:17.709853 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:17.709818    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:19.253810 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:19.253776    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" path="/var/lib/kubelet/pods/c98e2209-a2ee-430f-9d07-7f92d3e7d1ed/volumes"
Nov 03 21:32:19.861298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:19.861263    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:32:23.923774 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:23.923742    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:24.121553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:24.121478    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:24.318597 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:24.318513    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:25.933110 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:32:25.933077    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:32:34.274651 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:34.274613    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:34.472482 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:34.472447    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:40.679476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:40.679447    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:32:42.275362 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:42.272842    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:32:42.275362 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:42.272907    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt" podUID="30f93857-ee14-46af-a80e-7d9e242e2b73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:32:42.278936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:42.278872    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:42.605483 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:42.605443    5318 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:32:42.605796 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:42.605766    5318 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr" podUID="e69b42ce-7354-47e8-ac8f-22aeb8b0a70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:32:42.706970 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:42.706937    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:48.480146 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:48.480112    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:32:51.804930 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:32:51.804898    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:32:55.940286 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:32:55.940244    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:33:00.366021 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.365988    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"]
Nov 03 21:33:00.366605 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.366222    5318 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm" podUID="4a625db3-351f-421c-82f0-8850f8c3367c" containerName="manager" containerID="cri-o://fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936" gracePeriod=10
Nov 03 21:33:00.609919 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.609892    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 21:33:00.700425 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.700332    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume\") pod \"4a625db3-351f-421c-82f0-8850f8c3367c\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") "
Nov 03 21:33:00.700425 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.700393    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz\" (UniqueName: \"kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz\") pod \"4a625db3-351f-421c-82f0-8850f8c3367c\" (UID: \"4a625db3-351f-421c-82f0-8850f8c3367c\") "
Nov 03 21:33:00.700720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.700697    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume" pod "4a625db3-351f-421c-82f0-8850f8c3367c" (UID: "4a625db3-351f-421c-82f0-8850f8c3367c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:33:00.702440 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.702412    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz" pod "4a625db3-351f-421c-82f0-8850f8c3367c" (UID: "4a625db3-351f-421c-82f0-8850f8c3367c"). InnerVolumeSpecName "kube-api-access-r8pdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:33:00.802258 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.802219    5318 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a625db3-351f-421c-82f0-8850f8c3367c-extensions-socket-volume\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:33:00.802435 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:00.802263    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pdz\" (UniqueName: \"kubernetes.io/projected/4a625db3-351f-421c-82f0-8850f8c3367c-kube-api-access-r8pdz\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:33:01.361457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.361422    5318 generic.go:356] "Generic (PLEG): container finished" podID="4a625db3-351f-421c-82f0-8850f8c3367c" containerID="fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936" exitCode=0
Nov 03 21:33:01.361633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.361483    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"
Nov 03 21:33:01.361633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.361500    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm" event={"ID":"4a625db3-351f-421c-82f0-8850f8c3367c","Type":"ContainerDied","Data":"fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936"}
Nov 03 21:33:01.361633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.361560    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm" event={"ID":"4a625db3-351f-421c-82f0-8850f8c3367c","Type":"ContainerDied","Data":"c1787c4a69f65c695b756721a5824aa50679d06d5a9dfd5917a71581350f8da8"}
Nov 03 21:33:01.361633 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.361581    5318 scope.go:122] "RemoveContainer" containerID="fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936"
Nov 03 21:33:01.373522 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.373307    5318 scope.go:122] "RemoveContainer" containerID="fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936"
Nov 03 21:33:01.373748 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:33:01.373585    5318 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936\": container with ID starting with fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936 not found: ID does not exist" containerID="fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936"
Nov 03 21:33:01.373748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.373609    5318 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936"} err="failed to get container status \"fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936\": rpc error: code = NotFound desc = could not find container \"fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936\": container with ID starting with fdc96c8780eb38af28e60046f14e146aaa8b6e9cecbff59fe5ef4d716aadf936 not found: ID does not exist"
Nov 03 21:33:01.561277 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.561219    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"]
Nov 03 21:33:01.607481 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:01.607456    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-twpbm"]
Nov 03 21:33:03.253507 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:03.253476    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a625db3-351f-421c-82f0-8850f8c3367c" path="/var/lib/kubelet/pods/4a625db3-351f-421c-82f0-8850f8c3367c/volumes"
Nov 03 21:33:25.932484 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:33:25.932409    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:33:34.596044 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:33:34.596005    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:33:55.932461 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:33:55.932421    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:34:03.823294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.823257    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:03.938327 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938296    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"]
Nov 03 21:34:03.938740 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938726    5318 cpu_manager.go:414] "containerMap: removing container" podUID="4a625db3-351f-421c-82f0-8850f8c3367c" containerName="manager"
Nov 03 21:34:03.938785 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938742    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a625db3-351f-421c-82f0-8850f8c3367c" containerName="manager"
Nov 03 21:34:03.938785 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938757    5318 cpu_manager.go:414] "containerMap: removing container" podUID="c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" containerName="revoke-keys"
Nov 03 21:34:03.938785 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938762    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" containerName="revoke-keys"
Nov 03 21:34:03.938884 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938835    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a625db3-351f-421c-82f0-8850f8c3367c" containerName="manager"
Nov 03 21:34:03.938884 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.938845    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98e2209-a2ee-430f-9d07-7f92d3e7d1ed" containerName="revoke-keys"
Nov 03 21:34:03.942082 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.942065    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:03.990995 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.990961    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jmjmh\""
Nov 03 21:34:03.991872 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.991850    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4abe529-55f5-4e84-81df-45f1f7e89746-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mttnr\" (UID: \"d4abe529-55f5-4e84-81df-45f1f7e89746\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:03.991954 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:03.991888    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjv2\" (UniqueName: \"kubernetes.io/projected/d4abe529-55f5-4e84-81df-45f1f7e89746-kube-api-access-cfjv2\") pod \"kuadrant-operator-controller-manager-678d87766f-mttnr\" (UID: \"d4abe529-55f5-4e84-81df-45f1f7e89746\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:04.038985 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.038954    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"]
Nov 03 21:34:04.092973 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.092940    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjv2\" (UniqueName: \"kubernetes.io/projected/d4abe529-55f5-4e84-81df-45f1f7e89746-kube-api-access-cfjv2\") pod \"kuadrant-operator-controller-manager-678d87766f-mttnr\" (UID: \"d4abe529-55f5-4e84-81df-45f1f7e89746\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:04.093145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.093056    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4abe529-55f5-4e84-81df-45f1f7e89746-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mttnr\" (UID: \"d4abe529-55f5-4e84-81df-45f1f7e89746\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:04.093457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.093440    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4abe529-55f5-4e84-81df-45f1f7e89746-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mttnr\" (UID: \"d4abe529-55f5-4e84-81df-45f1f7e89746\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:04.150293 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.150254    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjv2\" (UniqueName: \"kubernetes.io/projected/d4abe529-55f5-4e84-81df-45f1f7e89746-kube-api-access-cfjv2\") pod \"kuadrant-operator-controller-manager-678d87766f-mttnr\" (UID: \"d4abe529-55f5-4e84-81df-45f1f7e89746\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:04.255412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.255373    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:04.715111 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:04.715089    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"]
Nov 03 21:34:04.718349 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:34:04.718299    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4abe529_55f5_4e84_81df_45f1f7e89746.slice/crio-f6367a0b3422578d96f70f45754025711e17f1144836450c1f5172b34d04fad4 WatchSource:0}: Error finding container f6367a0b3422578d96f70f45754025711e17f1144836450c1f5172b34d04fad4: Status 404 returned error can't find the container with id f6367a0b3422578d96f70f45754025711e17f1144836450c1f5172b34d04fad4
Nov 03 21:34:05.666689 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:05.666644    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr" event={"ID":"d4abe529-55f5-4e84-81df-45f1f7e89746","Type":"ContainerStarted","Data":"8d15688195fe9cb970d30d54738297f592a7fd233cbb99e98405b9311c2f30bc"}
Nov 03 21:34:05.667287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:05.666715    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr" event={"ID":"d4abe529-55f5-4e84-81df-45f1f7e89746","Type":"ContainerStarted","Data":"f6367a0b3422578d96f70f45754025711e17f1144836450c1f5172b34d04fad4"}
Nov 03 21:34:05.667287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:05.666782    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:05.870782 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:05.870729    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr" podStartSLOduration=2.870715002 podStartE2EDuration="2.870715002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:05.823856597 +0000 UTC m=+3838.803413436" watchObservedRunningTime="2026-11-03 21:34:05.870715002 +0000 UTC m=+3838.850271841"
Nov 03 21:34:16.674208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:16.674175    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mttnr"
Nov 03 21:34:23.872935 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:23.872900    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:34:25.959739 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:34:25.959705    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:34:34.945011 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:34.944981    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:38.173375 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:38.173347    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:34:41.465781 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:41.465745    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:34:50.470403 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:34:50.470316    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:34:55.932603 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:34:55.932566    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:35:07.861328 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:07.861302    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:35:07.870697 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:07.870678    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:35:10.138106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:10.138075    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:10.359496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:10.359463    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:35:10.807446 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:10.807412    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:11.003953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:11.003925    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:15.164188 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:15.164156    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:15.364987 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:15.364957    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:18.452873 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:18.452837    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:18.661645 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:18.661612    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:21.754607 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:21.754573    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:21.954555 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:21.954526    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:22.152371 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:22.152319    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:25.934376 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:35:25.934325    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:35:26.900161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:26.900130    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:29.496787 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:29.496751    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:35:31.958056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:31.958018    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:35.276542 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:35.276504    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:37.016791 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:37.016759    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:39.976756 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:39.976728    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:40.564349 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:40.564306    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:40.769814 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:40.769784    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:42.205912 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:42.205877    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:43.313331 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:43.313292    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:47.112885 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:47.112853    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:47.856940 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:47.856906    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:35:52.583476 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:52.583436    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:35:55.933043 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:35:55.933005    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:35:57.524551 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:35:57.524521    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:02.533324 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:02.533289    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:03.590670 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:03.590593    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:03.802216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:03.802178    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:07.413936 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:07.413897    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:12.820215 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:12.820166    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:18.218496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:18.218424    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:21.259086 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:21.259057    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:36:23.333095 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:23.333062    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:25.933181 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:36:25.933139    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:36:28.310735 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:28.310702    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:33.408187 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:33.408156    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:35.631097 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:35.631067    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:38.408352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:38.408313    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:43.415314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:43.415275    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:44.155616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.155580    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"]
Nov 03 21:36:44.159160 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.159138    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.207878 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.207851    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:36:44.253788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.253746    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:36:44.253788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.253752    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:36:44.253978 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.253808    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:36:44.255565 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.255541    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"]
Nov 03 21:36:44.336676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.336649    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.336676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.336680    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzcl\" (UniqueName: \"kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.336850 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.336716    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.437551 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.437473    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.437923 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.437578    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.437923 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.437600    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzcl\" (UniqueName: \"kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.438205 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.438186    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.439938 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.439914    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.495202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.495175    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzcl\" (UniqueName: \"kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl\") pod \"maas-api-revoke-sub-91ae994f-mtl42\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:44.772158 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:44.772084    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:45.138860 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:45.138823    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"]
Nov 03 21:36:45.141272 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:36:45.141238    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33282520_c751_4653_a024_8b21afb1320c.slice/crio-978c59ff6cb21e57b1cbaa4ef7bd4436989dd7838564e6c28b059df1af67c4ae WatchSource:0}: Error finding container 978c59ff6cb21e57b1cbaa4ef7bd4436989dd7838564e6c28b059df1af67c4ae: Status 404 returned error can't find the container with id 978c59ff6cb21e57b1cbaa4ef7bd4436989dd7838564e6c28b059df1af67c4ae
Nov 03 21:36:45.497913 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:45.497872    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42" event={"ID":"33282520-c751-4653-a024-8b21afb1320c","Type":"ContainerStarted","Data":"a6ac9643bf9fe61809526af61c48caeaca3844e0dbd327d7067be7c6e1f59ef1"}
Nov 03 21:36:45.498293 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:45.497956    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42" event={"ID":"33282520-c751-4653-a024-8b21afb1320c","Type":"ContainerStarted","Data":"978c59ff6cb21e57b1cbaa4ef7bd4436989dd7838564e6c28b059df1af67c4ae"}
Nov 03 21:36:45.607136 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:45.607081    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42" podStartSLOduration=1.607066316 podStartE2EDuration="1.607066316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:45.605330699 +0000 UTC m=+3998.584887564" watchObservedRunningTime="2026-11-03 21:36:45.607066316 +0000 UTC m=+3998.586623155"
Nov 03 21:36:46.504764 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:46.504722    5318 generic.go:356] "Generic (PLEG): container finished" podID="33282520-c751-4653-a024-8b21afb1320c" containerID="a6ac9643bf9fe61809526af61c48caeaca3844e0dbd327d7067be7c6e1f59ef1" exitCode=0
Nov 03 21:36:46.505270 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:46.504802    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42" event={"ID":"33282520-c751-4653-a024-8b21afb1320c","Type":"ContainerDied","Data":"a6ac9643bf9fe61809526af61c48caeaca3844e0dbd327d7067be7c6e1f59ef1"}
Nov 03 21:36:47.649190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.649165    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:47.770322 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.770247    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca\") pod \"33282520-c751-4653-a024-8b21afb1320c\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") "
Nov 03 21:36:47.770480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.770389    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca\") pod \"33282520-c751-4653-a024-8b21afb1320c\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") "
Nov 03 21:36:47.770480 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.770434    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl\" (UniqueName: \"kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl\") pod \"33282520-c751-4653-a024-8b21afb1320c\" (UID: \"33282520-c751-4653-a024-8b21afb1320c\") "
Nov 03 21:36:47.770792 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.770764    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca" pod "33282520-c751-4653-a024-8b21afb1320c" (UID: "33282520-c751-4653-a024-8b21afb1320c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:36:47.772420 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.772396    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl" pod "33282520-c751-4653-a024-8b21afb1320c" (UID: "33282520-c751-4653-a024-8b21afb1320c"). InnerVolumeSpecName "kube-api-access-bkzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:36:47.772420 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.772403    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca" pod "33282520-c751-4653-a024-8b21afb1320c" (UID: "33282520-c751-4653-a024-8b21afb1320c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:36:47.871993 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.871959    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzcl\" (UniqueName: \"kubernetes.io/projected/33282520-c751-4653-a024-8b21afb1320c-kube-api-access-bkzcl\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:36:47.871993 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.871989    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33282520-c751-4653-a024-8b21afb1320c-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:36:47.871993 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:47.872000    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33282520-c751-4653-a024-8b21afb1320c-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:36:48.421567 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:48.421533    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:48.517039 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:48.517004    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42" event={"ID":"33282520-c751-4653-a024-8b21afb1320c","Type":"ContainerDied","Data":"978c59ff6cb21e57b1cbaa4ef7bd4436989dd7838564e6c28b059df1af67c4ae"}
Nov 03 21:36:48.517039 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:48.517034    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"
Nov 03 21:36:48.517311 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:48.517061    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="978c59ff6cb21e57b1cbaa4ef7bd4436989dd7838564e6c28b059df1af67c4ae"
Nov 03 21:36:53.044547 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:53.044511    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"]
Nov 03 21:36:53.067302 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:53.067275    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc94ac84-vgdkq"]
Nov 03 21:36:53.254450 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:53.254418    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d719e26-ce33-45e0-9dbd-db135cbf73bb" path="/var/lib/kubelet/pods/2d719e26-ce33-45e0-9dbd-db135cbf73bb/volumes"
Nov 03 21:36:53.427910 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:53.427880    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:55.395575 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.395543    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:55.887734 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.887702    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"]
Nov 03 21:36:55.888116 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.888104    5318 cpu_manager.go:414] "containerMap: removing container" podUID="33282520-c751-4653-a024-8b21afb1320c" containerName="revoke-keys"
Nov 03 21:36:55.888166 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.888117    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33282520-c751-4653-a024-8b21afb1320c" containerName="revoke-keys"
Nov 03 21:36:55.888202 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.888180    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="33282520-c751-4653-a024-8b21afb1320c" containerName="revoke-keys"
Nov 03 21:36:55.891533 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.891512    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:55.933032 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:36:55.933003    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:36:55.946890 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.946866    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:55.947005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.946901    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjd4j\" (UniqueName: \"kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:55.947005 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.946926    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:55.986532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.986506    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:36:55.986532 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.986518    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:36:55.986718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.986574    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:36:55.986718 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.986611    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\""
Nov 03 21:36:55.988261 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:55.988242    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"]
Nov 03 21:36:56.047828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.047799    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.047975 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.047838    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjd4j\" (UniqueName: \"kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.047975 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.047867    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.048422 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.048405    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.050079 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.050060    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.105720 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.105692    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjd4j\" (UniqueName: \"kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.203768 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.203697    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:56.603591 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:56.603559    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"]
Nov 03 21:36:56.607346 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:36:56.607306    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8868944b_f94a_47b1_a36b_ad17afc481f8.slice/crio-5a9bf81ba1bbf40532c817677a6ca899d07783923e8543dd67d795e3a274b21a WatchSource:0}: Error finding container 5a9bf81ba1bbf40532c817677a6ca899d07783923e8543dd67d795e3a274b21a: Status 404 returned error can't find the container with id 5a9bf81ba1bbf40532c817677a6ca899d07783923e8543dd67d795e3a274b21a
Nov 03 21:36:57.106048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:57.106018    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:36:57.561705 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:57.561671    5318 generic.go:356] "Generic (PLEG): container finished" podID="8868944b-f94a-47b1-a36b-ad17afc481f8" containerID="dc9e0e49ae13b384888eb5ce3c15373e53518780387eba866a9e02b75b1b9e45" exitCode=0
Nov 03 21:36:57.561872 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:57.561761    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj" event={"ID":"8868944b-f94a-47b1-a36b-ad17afc481f8","Type":"ContainerDied","Data":"dc9e0e49ae13b384888eb5ce3c15373e53518780387eba866a9e02b75b1b9e45"}
Nov 03 21:36:57.561872 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:57.561818    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj" event={"ID":"8868944b-f94a-47b1-a36b-ad17afc481f8","Type":"ContainerStarted","Data":"5a9bf81ba1bbf40532c817677a6ca899d07783923e8543dd67d795e3a274b21a"}
Nov 03 21:36:58.640089 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.640022    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:36:58.699245 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.699215    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:58.776359 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.776310    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca\") pod \"8868944b-f94a-47b1-a36b-ad17afc481f8\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") "
Nov 03 21:36:58.776526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.776445    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca\") pod \"8868944b-f94a-47b1-a36b-ad17afc481f8\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") "
Nov 03 21:36:58.776526 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.776494    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j\" (UniqueName: \"kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j\") pod \"8868944b-f94a-47b1-a36b-ad17afc481f8\" (UID: \"8868944b-f94a-47b1-a36b-ad17afc481f8\") "
Nov 03 21:36:58.776685 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.776659    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca" pod "8868944b-f94a-47b1-a36b-ad17afc481f8" (UID: "8868944b-f94a-47b1-a36b-ad17afc481f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:36:58.776836 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.776820    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:36:58.778479 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.778460    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j" pod "8868944b-f94a-47b1-a36b-ad17afc481f8" (UID: "8868944b-f94a-47b1-a36b-ad17afc481f8"). InnerVolumeSpecName "kube-api-access-fjd4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:36:58.778661 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.778642    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca" pod "8868944b-f94a-47b1-a36b-ad17afc481f8" (UID: "8868944b-f94a-47b1-a36b-ad17afc481f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:36:58.878007 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.877974    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8868944b-f94a-47b1-a36b-ad17afc481f8-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:36:58.878007 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:58.878004    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjd4j\" (UniqueName: \"kubernetes.io/projected/8868944b-f94a-47b1-a36b-ad17afc481f8-kube-api-access-fjd4j\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:36:59.572397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:59.572304    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj" event={"ID":"8868944b-f94a-47b1-a36b-ad17afc481f8","Type":"ContainerDied","Data":"5a9bf81ba1bbf40532c817677a6ca899d07783923e8543dd67d795e3a274b21a"}
Nov 03 21:36:59.572397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:59.572327    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"
Nov 03 21:36:59.572397 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:36:59.572369    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a9bf81ba1bbf40532c817677a6ca899d07783923e8543dd67d795e3a274b21a"
Nov 03 21:37:00.425224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:00.425180    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:01.156404 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:01.156321    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:01.347421 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:01.347384    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:03.810182 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:03.810145    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:07.849120 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:07.848693    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:08.048316 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:08.048284    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:08.724972 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:08.724938    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:13.064215 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:13.064152    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:14.675401 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:14.675371    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:14.967524 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:14.967430    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:15.243211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:15.243133    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"]
Nov 03 21:37:15.289360 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:15.289311    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7b11dd-j5wrj"]
Nov 03 21:37:15.834708 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:15.834672    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:17.254781 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:17.254749    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8868944b-f94a-47b1-a36b-ad17afc481f8" path="/var/lib/kubelet/pods/8868944b-f94a-47b1-a36b-ad17afc481f8/volumes"
Nov 03 21:37:17.563569 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:17.563491    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:37:21.844681 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:21.844640    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:22.053193 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:22.053161    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:22.269793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:22.269703    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:25.933260 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:37:25.933222    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:37:35.959850 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:35.959812    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:37.058957 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:37.058927    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:38.007117 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:38.007087    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:40.726916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:40.726875    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:43.760308 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:43.760273    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:45.263087 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:45.263051    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:37:53.229793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:37:53.229759    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:37:55.932815 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:37:55.932777    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:38:07.100777 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:07.100736    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:08.543957 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:08.543920    5318 scope.go:122] "RemoveContainer" containerID="1ab5f88912d3b28998d773743ee3b1e3724638216bef8565976c6b1ba71dc046"
Nov 03 21:38:08.552959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:08.552944    5318 scope.go:122] "RemoveContainer" containerID="cf23de6df7d6723159ddc92704837e0599b34c252751755cf43e1e134b63421d"
Nov 03 21:38:11.755759 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:11.755729    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:38:25.933456 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:38:25.933420    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:38:35.697872 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:35.697840    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:35.893225 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:35.893179    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:36.280412 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:36.280386    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:43.175248 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:43.175210    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:38:48.194129 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:48.194092    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:54.863817 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:54.863777    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:38:55.268203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:38:55.268129    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:38:55.936506 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:38:55.936466    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:39:07.084639 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:07.084599    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:39:08.048320 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:08.048277    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:39:09.021106 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:09.021059    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:39:19.679611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:19.679571    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:39:25.933631 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:39:25.933589    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:39:31.149256 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:31.149217    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:39:36.615003 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:36.614963    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:39:54.487616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:39:54.487578    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:39:55.932376 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:39:55.932315    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:40:01.506107 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:01.506039    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:02.159230 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:02.159158    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:02.378867 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:02.378818    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:05.705597 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:05.705565    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:05.924803 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:05.924767    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:07.895224 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:07.895196    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:40:07.905629 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:07.905610    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:40:09.045303 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:09.045272    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:09.247065 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:09.247025    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:11.305394 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:11.305353    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:11.506834 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:11.506781    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:11.713825 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:11.713785    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:16.726272 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:16.726234    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:19.254983 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:19.254946    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:40:21.832248 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:21.832198    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:25.933727 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:40:25.933689    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:40:26.812317 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:26.812283    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:32.306691 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:32.306633    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:36.955314 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:36.955275    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:38.507697 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:38.507664    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:42.466860 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:42.466830    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:47.925186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:47.925104    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:52.124947 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.124914    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"]
Nov 03 21:40:52.125409 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.125293    5318 cpu_manager.go:414] "containerMap: removing container" podUID="8868944b-f94a-47b1-a36b-ad17afc481f8" containerName="revoke-keys"
Nov 03 21:40:52.125409 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.125303    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8868944b-f94a-47b1-a36b-ad17afc481f8" containerName="revoke-keys"
Nov 03 21:40:52.125519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.125411    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="8868944b-f94a-47b1-a36b-ad17afc481f8" containerName="revoke-keys"
Nov 03 21:40:52.128208 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.128188    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.176807 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.176776    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:40:52.223115 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.223089    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k69r\""
Nov 03 21:40:52.223281 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.223139    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:40:52.223281 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.223230    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:40:52.224943 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.224924    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"]
Nov 03 21:40:52.269403 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.269369    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb9v2\" (UniqueName: \"kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.269552 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.269432    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.269552 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.269526    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.370223 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.370186    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb9v2\" (UniqueName: \"kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.370473 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.370246    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.370473 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.370315    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.370960 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.370934    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.372749 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.372725    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.427733 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.427662    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb9v2\" (UniqueName: \"kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2\") pod \"maas-api-revoke-keys-e2e-ait-4f1680df-ztst6\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.440441 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.440414    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:52.803203 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.803177    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"]
Nov 03 21:40:52.806178 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:40:52.806146    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a299367_181e_4748_8c05_0c9434865439.slice/crio-3a4747121da4232eae9ccf879b2ad53fb316b88103879d42e8b905423442de51 WatchSource:0}: Error finding container 3a4747121da4232eae9ccf879b2ad53fb316b88103879d42e8b905423442de51: Status 404 returned error can't find the container with id 3a4747121da4232eae9ccf879b2ad53fb316b88103879d42e8b905423442de51
Nov 03 21:40:52.828365 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.828325    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6" event={"ID":"3a299367-181e-4748-8c05-0c9434865439","Type":"ContainerStarted","Data":"3a4747121da4232eae9ccf879b2ad53fb316b88103879d42e8b905423442de51"}
Nov 03 21:40:52.926879 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:52.926846    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:40:53.835924 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:53.835844    5318 generic.go:356] "Generic (PLEG): container finished" podID="3a299367-181e-4748-8c05-0c9434865439" containerID="ba414b45a80107171c63edf61596eca5fd621c21d258341c0a83d1e8a5ff352b" exitCode=0
Nov 03 21:40:53.835924 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:53.835898    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6" event={"ID":"3a299367-181e-4748-8c05-0c9434865439","Type":"ContainerDied","Data":"ba414b45a80107171c63edf61596eca5fd621c21d258341c0a83d1e8a5ff352b"}
Nov 03 21:40:54.972676 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.972649    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:54.996051 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.996019    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2\" (UniqueName: \"kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2\") pod \"3a299367-181e-4748-8c05-0c9434865439\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") "
Nov 03 21:40:54.996183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.996076    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca\") pod \"3a299367-181e-4748-8c05-0c9434865439\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") "
Nov 03 21:40:54.996183 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.996122    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca\") pod \"3a299367-181e-4748-8c05-0c9434865439\" (UID: \"3a299367-181e-4748-8c05-0c9434865439\") "
Nov 03 21:40:54.996512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.996477    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca" pod "3a299367-181e-4748-8c05-0c9434865439" (UID: "3a299367-181e-4748-8c05-0c9434865439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:40:54.998167 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.998145    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2" pod "3a299367-181e-4748-8c05-0c9434865439" (UID: "3a299367-181e-4748-8c05-0c9434865439"). InnerVolumeSpecName "kube-api-access-qb9v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:40:54.998280 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:54.998186    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca" pod "3a299367-181e-4748-8c05-0c9434865439" (UID: "3a299367-181e-4748-8c05-0c9434865439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:40:55.097298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:55.097220    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb9v2\" (UniqueName: \"kubernetes.io/projected/3a299367-181e-4748-8c05-0c9434865439-kube-api-access-qb9v2\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:55.097298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:55.097245    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a299367-181e-4748-8c05-0c9434865439-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:55.097298 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:55.097255    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a299367-181e-4748-8c05-0c9434865439-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:40:55.847399 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:55.847368    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"
Nov 03 21:40:55.847589 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:55.847394    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6" event={"ID":"3a299367-181e-4748-8c05-0c9434865439","Type":"ContainerDied","Data":"3a4747121da4232eae9ccf879b2ad53fb316b88103879d42e8b905423442de51"}
Nov 03 21:40:55.847589 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:55.847469    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a4747121da4232eae9ccf879b2ad53fb316b88103879d42e8b905423442de51"
Nov 03 21:40:55.931697 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:40:55.931660    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:40:57.812674 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:40:57.812643    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:02.802176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:02.802149    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:04.073865 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:04.073814    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:04.285543 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:04.285507    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:07.641734 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:07.641703    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:12.047543 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:12.047509    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:13.679863 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:13.679830    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:13.984979 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:13.984896    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:14.121921 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:14.121888    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"]
Nov 03 21:41:14.167960 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:14.167921    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4f1680df-ztst6"]
Nov 03 21:41:14.810834 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:14.810801    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:15.020935 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:15.020904    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:15.253634 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:15.253599    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a299367-181e-4748-8c05-0c9434865439" path="/var/lib/kubelet/pods/3a299367-181e-4748-8c05-0c9434865439/volumes"
Nov 03 21:41:16.246056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:16.246024    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:41:18.415199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:18.415168    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:19.724324 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:19.724287    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:20.319860 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:20.319812    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:20.522045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:20.522003    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:25.931849 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:41:25.931812    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:41:32.876578 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:32.876540    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:33.082064 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:33.082031    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:36.211903 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:36.211870    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:36.417206 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:36.417175    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:37.284586 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:37.284543    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:39.375647 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:39.375611    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:39.582670 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:39.582635    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:44.418585 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:44.418550    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:47.169499 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:47.169466    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:41:48.089932 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:48.089896    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"]
Nov 03 21:41:48.091572 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:48.091545    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ae994f-mtl42"]
Nov 03 21:41:49.253841 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:49.253801    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33282520-c751-4653-a024-8b21afb1320c" path="/var/lib/kubelet/pods/33282520-c751-4653-a024-8b21afb1320c/volumes"
Nov 03 21:41:49.433566 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:49.433535    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:54.316512 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:54.316482    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:55.932996 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:41:55.932962    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:41:57.683450 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:57.683412    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:41:59.721741 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:41:59.721709    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:04.621549 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:04.621514    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:09.615961 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:09.615930    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:14.575426 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:14.575395    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:19.532108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:19.532034    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:24.415299 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:24.415265    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:25.932852 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:42:25.932820    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:42:29.814593 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:29.814560    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:34.801929 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:34.801879    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:35.155747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:35.155712    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:35.436535 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:35.436458    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:37.604218 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:37.604158    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:39.802864 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:39.802832    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:44.814162 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:44.814127    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:49.810637 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:49.810592    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:54.727228 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:54.727195    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:55.932539 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:42:55.932505    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:42:59.281446 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:59.281410    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:42:59.800917 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:42:59.800887    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:04.718745 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:04.718714    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:08.608212 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:08.608183    5318 scope.go:122] "RemoveContainer" containerID="dc9e0e49ae13b384888eb5ce3c15373e53518780387eba866a9e02b75b1b9e45"
Nov 03 21:43:08.616883 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:08.616863    5318 scope.go:122] "RemoveContainer" containerID="a6ac9643bf9fe61809526af61c48caeaca3844e0dbd327d7067be7c6e1f59ef1"
Nov 03 21:43:09.727559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:09.727523    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:14.634699 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:14.634663    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:17.950419 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:17.950383    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"]
Nov 03 21:43:17.950807 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:17.950765    5318 cpu_manager.go:414] "containerMap: removing container" podUID="3a299367-181e-4748-8c05-0c9434865439" containerName="revoke-keys"
Nov 03 21:43:17.950807 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:17.950776    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a299367-181e-4748-8c05-0c9434865439" containerName="revoke-keys"
Nov 03 21:43:17.950881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:17.950869    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a299367-181e-4748-8c05-0c9434865439" containerName="revoke-keys"
Nov 03 21:43:17.953895 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:17.953874    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.002493 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.002472    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:43:18.048562 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.048531    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:43:18.048818 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.048798    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:43:18.049028 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.049012    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k69r\""
Nov 03 21:43:18.050643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.050619    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"]
Nov 03 21:43:18.059206 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.059187    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.059264 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.059248    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.059390 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.059373    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5768\" (UniqueName: \"kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.160258 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.160224    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.160432 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.160284    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5768\" (UniqueName: \"kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.160432 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.160396    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.160870 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.160853    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.162611 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.162592    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.217456 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.217401    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5768\" (UniqueName: \"kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768\") pod \"maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.267467 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.267444    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:18.631747 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:18.631726    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"]
Nov 03 21:43:18.634673 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:43:18.634643    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda960ae71_095e_4b8b_b8d8_c901b3f36509.slice/crio-ee6f661e7ebcab18eb5b837f1e8f121e3be5346e3bbfe745d2ce6f7b1f32e07e WatchSource:0}: Error finding container ee6f661e7ebcab18eb5b837f1e8f121e3be5346e3bbfe745d2ce6f7b1f32e07e: Status 404 returned error can't find the container with id ee6f661e7ebcab18eb5b837f1e8f121e3be5346e3bbfe745d2ce6f7b1f32e07e
Nov 03 21:43:19.543626 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:19.543596    5318 generic.go:356] "Generic (PLEG): container finished" podID="a960ae71-095e-4b8b-b8d8-c901b3f36509" containerID="6c5bca1c5cf94255b7f449f4c737b10591ceb4eae34105367618618761e8462c" exitCode=0
Nov 03 21:43:19.544008 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:19.543689    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs" event={"ID":"a960ae71-095e-4b8b-b8d8-c901b3f36509","Type":"ContainerDied","Data":"6c5bca1c5cf94255b7f449f4c737b10591ceb4eae34105367618618761e8462c"}
Nov 03 21:43:19.544008 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:19.543760    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs" event={"ID":"a960ae71-095e-4b8b-b8d8-c901b3f36509","Type":"ContainerStarted","Data":"ee6f661e7ebcab18eb5b837f1e8f121e3be5346e3bbfe745d2ce6f7b1f32e07e"}
Nov 03 21:43:19.842061 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:19.842032    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:20.679215 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.679191    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:20.785852 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.785827    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768\" (UniqueName: \"kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768\") pod \"a960ae71-095e-4b8b-b8d8-c901b3f36509\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") "
Nov 03 21:43:20.786006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.785888    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca\") pod \"a960ae71-095e-4b8b-b8d8-c901b3f36509\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") "
Nov 03 21:43:20.786006 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.785913    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca\") pod \"a960ae71-095e-4b8b-b8d8-c901b3f36509\" (UID: \"a960ae71-095e-4b8b-b8d8-c901b3f36509\") "
Nov 03 21:43:20.786300 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.786270    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca" pod "a960ae71-095e-4b8b-b8d8-c901b3f36509" (UID: "a960ae71-095e-4b8b-b8d8-c901b3f36509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:43:20.787917 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.787889    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768" pod "a960ae71-095e-4b8b-b8d8-c901b3f36509" (UID: "a960ae71-095e-4b8b-b8d8-c901b3f36509"). InnerVolumeSpecName "kube-api-access-d5768". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:20.788068 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.787946    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca" pod "a960ae71-095e-4b8b-b8d8-c901b3f36509" (UID: "a960ae71-095e-4b8b-b8d8-c901b3f36509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:43:20.887622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.887592    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5768\" (UniqueName: \"kubernetes.io/projected/a960ae71-095e-4b8b-b8d8-c901b3f36509-kube-api-access-d5768\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:20.887622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.887620    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:20.887809 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:20.887630    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a960ae71-095e-4b8b-b8d8-c901b3f36509-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:43:21.554511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:21.554435    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"
Nov 03 21:43:21.554511 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:21.554469    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs" event={"ID":"a960ae71-095e-4b8b-b8d8-c901b3f36509","Type":"ContainerDied","Data":"ee6f661e7ebcab18eb5b837f1e8f121e3be5346e3bbfe745d2ce6f7b1f32e07e"}
Nov 03 21:43:21.554692 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:21.554513    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee6f661e7ebcab18eb5b837f1e8f121e3be5346e3bbfe745d2ce6f7b1f32e07e"
Nov 03 21:43:25.154579 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:25.154534    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:25.932611 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:43:25.932568    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:43:29.989773 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:29.989737    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:30.224282 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:30.224252    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:37.937872 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:37.937839    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:39.472323 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:39.472283    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:41.050447 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:41.050412    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:41.359433 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:41.359401    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:41.625940 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:41.625858    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"]
Nov 03 21:43:41.672294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:41.672257    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-bbda3a-f89vs"]
Nov 03 21:43:42.110989 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:42.110957    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:43.253998 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:43.253964    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a960ae71-095e-4b8b-b8d8-c901b3f36509" path="/var/lib/kubelet/pods/a960ae71-095e-4b8b-b8d8-c901b3f36509/volumes"
Nov 03 21:43:43.887092 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:43.887057    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:43:48.026632 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:48.026559    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:48.273101 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:48.273069    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:48.487563 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:48.487535    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:48.800671 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:48.800585    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:52.680906 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:52.680868    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:52.895352 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:52.895302    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:43:55.933313 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:43:55.933276    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:43:56.890283 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:43:56.890238    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:44:07.557495 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:07.557461    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:44:15.839521 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.839484    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"]
Nov 03 21:44:15.840176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.840056    5318 cpu_manager.go:414] "containerMap: removing container" podUID="a960ae71-095e-4b8b-b8d8-c901b3f36509" containerName="revoke-keys"
Nov 03 21:44:15.840176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.840076    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a960ae71-095e-4b8b-b8d8-c901b3f36509" containerName="revoke-keys"
Nov 03 21:44:15.840305 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.840216    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="a960ae71-095e-4b8b-b8d8-c901b3f36509" containerName="revoke-keys"
Nov 03 21:44:15.842921 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.842900    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.887762 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.887738    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.887861 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.887779    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8zk\" (UniqueName: \"kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.887861 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.887801    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.891652 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.891633    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:44:15.937614 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.937590    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k69r\""
Nov 03 21:44:15.937614 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.937590    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:44:15.937786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.937676    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:44:15.939396 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.939377    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"]
Nov 03 21:44:15.988445 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.988410    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.988582 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.988548    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.988628 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.988602    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8zk\" (UniqueName: \"kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.989159 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.989143    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:15.990803 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:15.990785    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:16.045627 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:16.045601    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8zk\" (UniqueName: \"kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk\") pod \"maas-api-revoke-sub-f03629db-ffsl9\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:16.156094 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:16.156009    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:16.518618 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:16.518446    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"]
Nov 03 21:44:16.844726 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:16.844695    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9" event={"ID":"6b541c92-c60b-4345-a238-7c5bceccc120","Type":"ContainerStarted","Data":"413713cf7235856370771f872a545d10e5201601ec018bdcd3f399c6cdda099e"}
Nov 03 21:44:16.845119 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:16.844745    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9" event={"ID":"6b541c92-c60b-4345-a238-7c5bceccc120","Type":"ContainerStarted","Data":"0da1061dcf96ff95f12269b56018f9873b15e2c80f7757b44e2441ac3a4e60f8"}
Nov 03 21:44:16.953663 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:16.953612    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9" podStartSLOduration=1.95359842 podStartE2EDuration="1.95359842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:16.952730923 +0000 UTC m=+4449.932287747" watchObservedRunningTime="2026-11-03 21:44:16.95359842 +0000 UTC m=+4449.933155260"
Nov 03 21:44:17.850738 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:17.850709    5318 generic.go:356] "Generic (PLEG): container finished" podID="6b541c92-c60b-4345-a238-7c5bceccc120" containerID="413713cf7235856370771f872a545d10e5201601ec018bdcd3f399c6cdda099e" exitCode=0
Nov 03 21:44:17.851171 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:17.850804    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9" event={"ID":"6b541c92-c60b-4345-a238-7c5bceccc120","Type":"ContainerDied","Data":"413713cf7235856370771f872a545d10e5201601ec018bdcd3f399c6cdda099e"}
Nov 03 21:44:18.992048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:18.992023    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:19.015218 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.015193    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca\") pod \"6b541c92-c60b-4345-a238-7c5bceccc120\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") "
Nov 03 21:44:19.015384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.015230    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca\") pod \"6b541c92-c60b-4345-a238-7c5bceccc120\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") "
Nov 03 21:44:19.015384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.015306    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk\" (UniqueName: \"kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk\") pod \"6b541c92-c60b-4345-a238-7c5bceccc120\" (UID: \"6b541c92-c60b-4345-a238-7c5bceccc120\") "
Nov 03 21:44:19.015723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.015687    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca" pod "6b541c92-c60b-4345-a238-7c5bceccc120" (UID: "6b541c92-c60b-4345-a238-7c5bceccc120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:19.017394 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.017362    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca" pod "6b541c92-c60b-4345-a238-7c5bceccc120" (UID: "6b541c92-c60b-4345-a238-7c5bceccc120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:19.017559 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.017459    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk" pod "6b541c92-c60b-4345-a238-7c5bceccc120" (UID: "6b541c92-c60b-4345-a238-7c5bceccc120"). InnerVolumeSpecName "kube-api-access-7s8zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:19.116949 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.116870    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8zk\" (UniqueName: \"kubernetes.io/projected/6b541c92-c60b-4345-a238-7c5bceccc120-kube-api-access-7s8zk\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:19.116949 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.116897    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:19.116949 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.116908    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b541c92-c60b-4345-a238-7c5bceccc120-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:19.862023 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.861998    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"
Nov 03 21:44:19.862186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.862036    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9" event={"ID":"6b541c92-c60b-4345-a238-7c5bceccc120","Type":"ContainerDied","Data":"0da1061dcf96ff95f12269b56018f9873b15e2c80f7757b44e2441ac3a4e60f8"}
Nov 03 21:44:19.862186 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:19.862083    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0da1061dcf96ff95f12269b56018f9873b15e2c80f7757b44e2441ac3a4e60f8"
Nov 03 21:44:25.933283 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:44:25.933247    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:44:28.564233 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:28.564199    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:44:36.790169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.790130    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"]
Nov 03 21:44:36.790731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.790607    5318 cpu_manager.go:414] "containerMap: removing container" podUID="6b541c92-c60b-4345-a238-7c5bceccc120" containerName="revoke-keys"
Nov 03 21:44:36.790731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.790624    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b541c92-c60b-4345-a238-7c5bceccc120" containerName="revoke-keys"
Nov 03 21:44:36.790731 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.790718    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b541c92-c60b-4345-a238-7c5bceccc120" containerName="revoke-keys"
Nov 03 21:44:36.793854 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.793832    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:36.888729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.888700    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:44:36.888729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.888723    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:44:36.888729 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.888699    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:44:36.889055 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.888787    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k69r\""
Nov 03 21:44:36.890616 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.890594    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"]
Nov 03 21:44:36.922327 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.922302    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:36.922467 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.922432    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwp6l\" (UniqueName: \"kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:36.922519 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:36.922502    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.023937 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.023908    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.024100 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.024005    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.024143 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.024119    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwp6l\" (UniqueName: \"kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.024683 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.024663    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.026304 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.026284    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.081709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.081638    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwp6l\" (UniqueName: \"kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l\") pod \"maas-api-revoke-sub-0165d7d2-p9lgp\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.106709 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.106680    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:37.475163 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.475132    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"]
Nov 03 21:44:37.476562 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:44:37.476533    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde886ca3_921c_4662_868e_5005d3fc0ebb.slice/crio-0a86e9dc0b79df9be5f84780241b230baa110a16842c60cc7c9c11312a7fe61c WatchSource:0}: Error finding container 0a86e9dc0b79df9be5f84780241b230baa110a16842c60cc7c9c11312a7fe61c: Status 404 returned error can't find the container with id 0a86e9dc0b79df9be5f84780241b230baa110a16842c60cc7c9c11312a7fe61c
Nov 03 21:44:37.960299 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.960266    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp" event={"ID":"de886ca3-921c-4662-868e-5005d3fc0ebb","Type":"ContainerStarted","Data":"729ace9586c994ecced2f568f2e548e2721db1622d32e71a53d638d21935866a"}
Nov 03 21:44:37.960727 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:37.960316    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp" event={"ID":"de886ca3-921c-4662-868e-5005d3fc0ebb","Type":"ContainerStarted","Data":"0a86e9dc0b79df9be5f84780241b230baa110a16842c60cc7c9c11312a7fe61c"}
Nov 03 21:44:38.068667 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:38.068560    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp" podStartSLOduration=2.068541942 podStartE2EDuration="2.068541942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:38.067752121 +0000 UTC m=+4471.047308958" watchObservedRunningTime="2026-11-03 21:44:38.068541942 +0000 UTC m=+4471.048098783"
Nov 03 21:44:38.257430 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:38.257394    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:44:38.458089 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:38.458058    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:44:38.966924 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:38.966888    5318 generic.go:356] "Generic (PLEG): container finished" podID="de886ca3-921c-4662-868e-5005d3fc0ebb" containerID="729ace9586c994ecced2f568f2e548e2721db1622d32e71a53d638d21935866a" exitCode=0
Nov 03 21:44:38.967384 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:38.966969    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp" event={"ID":"de886ca3-921c-4662-868e-5005d3fc0ebb","Type":"ContainerDied","Data":"729ace9586c994ecced2f568f2e548e2721db1622d32e71a53d638d21935866a"}
Nov 03 21:44:40.103835 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.103806    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:40.257737 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.257656    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca\") pod \"de886ca3-921c-4662-868e-5005d3fc0ebb\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") "
Nov 03 21:44:40.257916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.257748    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l\" (UniqueName: \"kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l\") pod \"de886ca3-921c-4662-868e-5005d3fc0ebb\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") "
Nov 03 21:44:40.257916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.257785    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca\") pod \"de886ca3-921c-4662-868e-5005d3fc0ebb\" (UID: \"de886ca3-921c-4662-868e-5005d3fc0ebb\") "
Nov 03 21:44:40.258190 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.258166    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca" pod "de886ca3-921c-4662-868e-5005d3fc0ebb" (UID: "de886ca3-921c-4662-868e-5005d3fc0ebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:40.259793 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.259770    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l" pod "de886ca3-921c-4662-868e-5005d3fc0ebb" (UID: "de886ca3-921c-4662-868e-5005d3fc0ebb"). InnerVolumeSpecName "kube-api-access-kwp6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:40.259881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.259826    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca" pod "de886ca3-921c-4662-868e-5005d3fc0ebb" (UID: "de886ca3-921c-4662-868e-5005d3fc0ebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:40.358959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.358928    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwp6l\" (UniqueName: \"kubernetes.io/projected/de886ca3-921c-4662-868e-5005d3fc0ebb-kube-api-access-kwp6l\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:40.358959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.358956    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:40.358959 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.358966    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886ca3-921c-4662-868e-5005d3fc0ebb-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:40.979169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.979139    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"
Nov 03 21:44:40.979169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.979148    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp" event={"ID":"de886ca3-921c-4662-868e-5005d3fc0ebb","Type":"ContainerDied","Data":"0a86e9dc0b79df9be5f84780241b230baa110a16842c60cc7c9c11312a7fe61c"}
Nov 03 21:44:40.979457 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:40.979192    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a86e9dc0b79df9be5f84780241b230baa110a16842c60cc7c9c11312a7fe61c"
Nov 03 21:44:48.550308 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.550273    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"]
Nov 03 21:44:48.550828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.550714    5318 cpu_manager.go:414] "containerMap: removing container" podUID="de886ca3-921c-4662-868e-5005d3fc0ebb" containerName="revoke-keys"
Nov 03 21:44:48.550828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.550727    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de886ca3-921c-4662-868e-5005d3fc0ebb" containerName="revoke-keys"
Nov 03 21:44:48.550828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.550808    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="de886ca3-921c-4662-868e-5005d3fc0ebb" containerName="revoke-keys"
Nov 03 21:44:48.553986 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.553969    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.602664 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.602637    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:44:48.602821 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.602686    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:44:48.637586 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.637556    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.637586 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.637587    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.637828 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.637710    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tnv\" (UniqueName: \"kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.648622 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.648593    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:44:48.648758 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.648630    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k69r\""
Nov 03 21:44:48.650573 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.650553    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"]
Nov 03 21:44:48.738723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.738696    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.738723 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.738726    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.738939 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.738795    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tnv\" (UniqueName: \"kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.739372 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.739351    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.741073 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.741048    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.796401 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.796373    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tnv\" (UniqueName: \"kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv\") pod \"maas-api-revoke-sub-6b19338f-8bdfq\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:48.866788 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:48.866756    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:49.236810 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:49.236784    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"]
Nov 03 21:44:49.239678 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:44:49.239641    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47f1745_99da_457d_a6e8_2052c061589d.slice/crio-5360a3f9e34d1b3ccd404229cdc1bfffbc7130b2f7619ab06378deabf047ac03 WatchSource:0}: Error finding container 5360a3f9e34d1b3ccd404229cdc1bfffbc7130b2f7619ab06378deabf047ac03: Status 404 returned error can't find the container with id 5360a3f9e34d1b3ccd404229cdc1bfffbc7130b2f7619ab06378deabf047ac03
Nov 03 21:44:50.025383 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:50.025319    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq" event={"ID":"d47f1745-99da-457d-a6e8-2052c061589d","Type":"ContainerStarted","Data":"5a85c808d8a6439ad600c4912a93a0d9af266e25193ce0c48a830c76fa6d2bdd"}
Nov 03 21:44:50.025786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:50.025408    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq" event={"ID":"d47f1745-99da-457d-a6e8-2052c061589d","Type":"ContainerStarted","Data":"5360a3f9e34d1b3ccd404229cdc1bfffbc7130b2f7619ab06378deabf047ac03"}
Nov 03 21:44:50.134375 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:50.134302    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq" podStartSLOduration=2.13428686 podStartE2EDuration="2.13428686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:44:50.133131602 +0000 UTC m=+4483.112688444" watchObservedRunningTime="2026-11-03 21:44:50.13428686 +0000 UTC m=+4483.113843697"
Nov 03 21:44:51.032041 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:51.032007    5318 generic.go:356] "Generic (PLEG): container finished" podID="d47f1745-99da-457d-a6e8-2052c061589d" containerID="5a85c808d8a6439ad600c4912a93a0d9af266e25193ce0c48a830c76fa6d2bdd" exitCode=0
Nov 03 21:44:51.032470 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:51.032088    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq" event={"ID":"d47f1745-99da-457d-a6e8-2052c061589d","Type":"ContainerDied","Data":"5a85c808d8a6439ad600c4912a93a0d9af266e25193ce0c48a830c76fa6d2bdd"}
Nov 03 21:44:52.174866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.174838    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:52.268953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.268921    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv\" (UniqueName: \"kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv\") pod \"d47f1745-99da-457d-a6e8-2052c061589d\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") "
Nov 03 21:44:52.268953 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.268968    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca\") pod \"d47f1745-99da-457d-a6e8-2052c061589d\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") "
Nov 03 21:44:52.269169 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.269000    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca\") pod \"d47f1745-99da-457d-a6e8-2052c061589d\" (UID: \"d47f1745-99da-457d-a6e8-2052c061589d\") "
Nov 03 21:44:52.269373 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.269325    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca" pod "d47f1745-99da-457d-a6e8-2052c061589d" (UID: "d47f1745-99da-457d-a6e8-2052c061589d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:52.271176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.271150    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca" pod "d47f1745-99da-457d-a6e8-2052c061589d" (UID: "d47f1745-99da-457d-a6e8-2052c061589d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:52.271176 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.271159    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv" pod "d47f1745-99da-457d-a6e8-2052c061589d" (UID: "d47f1745-99da-457d-a6e8-2052c061589d"). InnerVolumeSpecName "kube-api-access-l5tnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:52.369952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.369913    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5tnv\" (UniqueName: \"kubernetes.io/projected/d47f1745-99da-457d-a6e8-2052c061589d-kube-api-access-l5tnv\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:52.369952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.369946    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-xks-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:52.369952 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:52.369957    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f1745-99da-457d-a6e8-2052c061589d-maas-api-openshift-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:44:53.050493 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:53.050464    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"
Nov 03 21:44:53.050659 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:53.050469    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq" event={"ID":"d47f1745-99da-457d-a6e8-2052c061589d","Type":"ContainerDied","Data":"5360a3f9e34d1b3ccd404229cdc1bfffbc7130b2f7619ab06378deabf047ac03"}
Nov 03 21:44:53.050659 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:44:53.050604    5318 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5360a3f9e34d1b3ccd404229cdc1bfffbc7130b2f7619ab06378deabf047ac03"
Nov 03 21:44:55.933875 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:44:55.933837    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:45:00.161108 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.161072    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"]
Nov 03 21:45:00.161521 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.161507    5318 cpu_manager.go:414] "containerMap: removing container" podUID="d47f1745-99da-457d-a6e8-2052c061589d" containerName="revoke-keys"
Nov 03 21:45:00.161588 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.161522    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47f1745-99da-457d-a6e8-2052c061589d" containerName="revoke-keys"
Nov 03 21:45:00.161626 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.161607    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47f1745-99da-457d-a6e8-2052c061589d" containerName="revoke-keys"
Nov 03 21:45:00.165125 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.165105    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.213823 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.213793    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:45:00.213823 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.213808    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:45:00.244356 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.244315    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-99k4b\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.244525 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.244490    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw546\" (UniqueName: \"kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546\") pod \"maas-api-key-cleanup-29895705-99k4b\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.260273 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.260252    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k69r\""
Nov 03 21:45:00.262056 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.262036    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"]
Nov 03 21:45:00.345597 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.345563    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw546\" (UniqueName: \"kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546\") pod \"maas-api-key-cleanup-29895705-99k4b\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.345797 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.345774    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-99k4b\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.346408 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.346384    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895705-99k4b\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.402684 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.402660    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw546\" (UniqueName: \"kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546\") pod \"maas-api-key-cleanup-29895705-99k4b\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.480786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.480695    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:00.846999 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:00.846973    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"]
Nov 03 21:45:00.849000 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:45:00.848970    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c57caf_9192_4d08_af4c_c1d07e5a3867.slice/crio-6d64791a16d3e14699f923203f6819ef9bb7c6c0c7021c12c504a0f5ac850fed WatchSource:0}: Error finding container 6d64791a16d3e14699f923203f6819ef9bb7c6c0c7021c12c504a0f5ac850fed: Status 404 returned error can't find the container with id 6d64791a16d3e14699f923203f6819ef9bb7c6c0c7021c12c504a0f5ac850fed
Nov 03 21:45:01.090226 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:01.090192    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" event={"ID":"31c57caf-9192-4d08-af4c-c1d07e5a3867","Type":"ContainerStarted","Data":"6d64791a16d3e14699f923203f6819ef9bb7c6c0c7021c12c504a0f5ac850fed"}
Nov 03 21:45:01.271658 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:01.271623    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-j6xpz"]
Nov 03 21:45:02.135146 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:02.135109    5318 generic.go:356] "Generic (PLEG): container finished" podID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerID="b650230c60c3436154dc6c787f9e07022a0a3d4ced5e0dd3d1e8887e5c53cf9a" exitCode=60
Nov 03 21:45:02.135547 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:02.135504    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" event={"ID":"31c57caf-9192-4d08-af4c-c1d07e5a3867","Type":"ContainerDied","Data":"b650230c60c3436154dc6c787f9e07022a0a3d4ced5e0dd3d1e8887e5c53cf9a"}
Nov 03 21:45:02.136423 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:02.136042    5318 scope.go:122] "RemoveContainer" containerID="b650230c60c3436154dc6c787f9e07022a0a3d4ced5e0dd3d1e8887e5c53cf9a"
Nov 03 21:45:03.144413 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:03.144373    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" event={"ID":"31c57caf-9192-4d08-af4c-c1d07e5a3867","Type":"ContainerStarted","Data":"84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139"}
Nov 03 21:45:03.257385 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:03.257320    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" podStartSLOduration=3.257307887 podStartE2EDuration="3.257307887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:03.255717446 +0000 UTC m=+4496.235274286" watchObservedRunningTime="2026-11-03 21:45:03.257307887 +0000 UTC m=+4496.236864728"
Nov 03 21:45:04.151077 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:04.151043    5318 generic.go:356] "Generic (PLEG): container finished" podID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerID="84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139" exitCode=60
Nov 03 21:45:04.151552 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:04.151114    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" event={"ID":"31c57caf-9192-4d08-af4c-c1d07e5a3867","Type":"ContainerDied","Data":"84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139"}
Nov 03 21:45:04.151552 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:04.151162    5318 scope.go:122] "RemoveContainer" containerID="b650230c60c3436154dc6c787f9e07022a0a3d4ced5e0dd3d1e8887e5c53cf9a"
Nov 03 21:45:04.151552 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:04.151443    5318 scope.go:122] "RemoveContainer" containerID="84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139"
Nov 03 21:45:04.151689 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:45:04.151670    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-99k4b_odh-ai-gateway-infra(31c57caf-9192-4d08-af4c-c1d07e5a3867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867"
Nov 03 21:45:05.157816 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:05.157788    5318 scope.go:122] "RemoveContainer" containerID="84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139"
Nov 03 21:45:05.158180 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:45:05.158004    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-99k4b_odh-ai-gateway-infra(31c57caf-9192-4d08-af4c-c1d07e5a3867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867"
Nov 03 21:45:07.933769 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:07.933743    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:45:07.943119 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:07.943092    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j68vs_8cdac693-b083-4a91-bd5c-ea56d23bd771/ovn-acl-logging/0.log"
Nov 03 21:45:15.248055 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:15.248026    5318 scope.go:122] "RemoveContainer" containerID="84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139"
Nov 03 21:45:16.217581 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:16.217549    5318 generic.go:356] "Generic (PLEG): container finished" podID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerID="6cf02f087d18fd32eedad2bc6d774f8ca21f7e276bae20a567eb4cd3332829ea" exitCode=60
Nov 03 21:45:16.217744 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:16.217623    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" event={"ID":"31c57caf-9192-4d08-af4c-c1d07e5a3867","Type":"ContainerDied","Data":"6cf02f087d18fd32eedad2bc6d774f8ca21f7e276bae20a567eb4cd3332829ea"}
Nov 03 21:45:16.217744 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:16.217682    5318 scope.go:122] "RemoveContainer" containerID="84a24d45fcf60cba4ac45bba744a6668f939578c95263c1b2dc76e52b9e5e139"
Nov 03 21:45:16.217983 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:16.217964    5318 scope.go:122] "RemoveContainer" containerID="6cf02f087d18fd32eedad2bc6d774f8ca21f7e276bae20a567eb4cd3332829ea"
Nov 03 21:45:16.218218 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:45:16.218199    5318 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895705-99k4b_odh-ai-gateway-infra(31c57caf-9192-4d08-af4c-c1d07e5a3867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867"
Nov 03 21:45:17.335575 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.335543    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"]
Nov 03 21:45:17.481577 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.481524    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:17.624072 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.624032    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546\" (UniqueName: \"kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546\") pod \"31c57caf-9192-4d08-af4c-c1d07e5a3867\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") "
Nov 03 21:45:17.624268 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.624102    5318 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca\") pod \"31c57caf-9192-4d08-af4c-c1d07e5a3867\" (UID: \"31c57caf-9192-4d08-af4c-c1d07e5a3867\") "
Nov 03 21:45:17.624507 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.624481    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca" pod "31c57caf-9192-4d08-af4c-c1d07e5a3867" (UID: "31c57caf-9192-4d08-af4c-c1d07e5a3867"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:17.626295 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.626265    5318 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546" pod "31c57caf-9192-4d08-af4c-c1d07e5a3867" (UID: "31c57caf-9192-4d08-af4c-c1d07e5a3867"). InnerVolumeSpecName "kube-api-access-rw546". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:17.725242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.725194    5318 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw546\" (UniqueName: \"kubernetes.io/projected/31c57caf-9192-4d08-af4c-c1d07e5a3867-kube-api-access-rw546\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:17.725242 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:17.725238    5318 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31c57caf-9192-4d08-af4c-c1d07e5a3867-maas-api-service-ca\") on node \"ip-10-0-135-141.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 21:45:18.230486 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:18.230447    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b" event={"ID":"31c57caf-9192-4d08-af4c-c1d07e5a3867","Type":"ContainerDied","Data":"6d64791a16d3e14699f923203f6819ef9bb7c6c0c7021c12c504a0f5ac850fed"}
Nov 03 21:45:18.230486 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:18.230478    5318 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"
Nov 03 21:45:18.230706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:18.230553    5318 scope.go:122] "RemoveContainer" containerID="6cf02f087d18fd32eedad2bc6d774f8ca21f7e276bae20a567eb4cd3332829ea"
Nov 03 21:45:18.383558 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:18.383525    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"]
Nov 03 21:45:18.428964 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:18.428932    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895705-99k4b"]
Nov 03 21:45:19.253297 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:19.253264    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" path="/var/lib/kubelet/pods/31c57caf-9192-4d08-af4c-c1d07e5a3867/volumes"
Nov 03 21:45:22.248145 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:22.248105    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bgkgw_4a1004ed-d52d-4a18-a74a-590f023c59f8/manager/0.log"
Nov 03 21:45:23.060178 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:23.060147    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5swbs_e65d8a64-1bbf-4e99-bed9-83307bcb8da6/manager/0.log"
Nov 03 21:45:23.938755 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:23.938714    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvk_de74614d-9ad6-4417-a716-0fc1375715b6/server/0.log"
Nov 03 21:45:24.163830 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:24.163799    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-tdb2j_eb6f8913-899f-4e7b-874a-7325937d52a9/manager/0.log"
Nov 03 21:45:25.843165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:25.843138    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-p9lgp_de886ca3-921c-4662-868e-5005d3fc0ebb/revoke-keys/0.log"
Nov 03 21:45:25.932300 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:45:25.932265    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:45:26.066561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:26.066530    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-8bdfq_d47f1745-99da-457d-a6e8-2052c061589d/revoke-keys/0.log"
Nov 03 21:45:26.284598 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:26.284523    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ffsl9_6b541c92-c60b-4345-a238-7c5bceccc120/revoke-keys/0.log"
Nov 03 21:45:29.037548 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:29.037521    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-llhxk_6218182d-10a0-4142-b413-46c95a9ecee8/kuadrant-console-plugin/0.log"
Nov 03 21:45:29.288180 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:29.288106    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mttnr_d4abe529-55f5-4e84-81df-45f1f7e89746/manager/0.log"
Nov 03 21:45:29.692311 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:29.692285    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j6xpz_ec3e62c6-7c98-4e0e-ad88-4653f9a1982e/limitador/0.log"
Nov 03 21:45:31.271422 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:31.271396    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ccb65679-h9rdf_62eb7e47-ff1b-43e1-8752-c16af0654016/kube-auth-proxy/0.log"
Nov 03 21:45:31.921164 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:31.921135    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt_30f93857-ee14-46af-a80e-7d9e242e2b73/istio-proxy/0.log"
Nov 03 21:45:32.468881 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:32.468851    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-cqqww_7534fda2-73dd-40a8-b44b-d8dc20f7d6c8/istio-proxy/0.log"
Nov 03 21:45:33.287045 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:33.287010    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr_e69b42ce-7354-47e8-ac8f-22aeb8b0a70a/istio-proxy/0.log"
Nov 03 21:45:38.896408 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:38.896374    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:45:47.878496 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:47.878460    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bgkgw_4a1004ed-d52d-4a18-a74a-590f023c59f8/manager/0.log"
Nov 03 21:45:48.467388 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:48.467360    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5swbs_e65d8a64-1bbf-4e99-bed9-83307bcb8da6/manager/0.log"
Nov 03 21:45:49.074180 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:49.074148    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvk_de74614d-9ad6-4417-a716-0fc1375715b6/server/0.log"
Nov 03 21:45:49.304007 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:49.303975    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-tdb2j_eb6f8913-899f-4e7b-874a-7325937d52a9/manager/0.log"
Nov 03 21:45:50.612046 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:50.612009    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-p9lgp_de886ca3-921c-4662-868e-5005d3fc0ebb/revoke-keys/0.log"
Nov 03 21:45:50.846893 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:50.846868    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-8bdfq_d47f1745-99da-457d-a6e8-2052c061589d/revoke-keys/0.log"
Nov 03 21:45:51.066717 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:51.066649    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ffsl9_6b541c92-c60b-4345-a238-7c5bceccc120/revoke-keys/0.log"
Nov 03 21:45:53.381325 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:53.381293    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-llhxk_6218182d-10a0-4142-b413-46c95a9ecee8/kuadrant-console-plugin/0.log"
Nov 03 21:45:53.617805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:53.617774    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mttnr_d4abe529-55f5-4e84-81df-45f1f7e89746/manager/0.log"
Nov 03 21:45:53.923887 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:53.923857    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j6xpz_ec3e62c6-7c98-4e0e-ad88-4653f9a1982e/limitador/0.log"
Nov 03 21:45:55.278247 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:55.278218    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ccb65679-h9rdf_62eb7e47-ff1b-43e1-8752-c16af0654016/kube-auth-proxy/0.log"
Nov 03 21:45:55.932753 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:45:55.932715    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:45:55.952805 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:55.952785    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-bkwrt_30f93857-ee14-46af-a80e-7d9e242e2b73/istio-proxy/0.log"
Nov 03 21:45:56.954903 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:56.954871    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:45:57.161199 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:45:57.161168    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:46:06.562423 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:46:06.562390    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-cqqww_7534fda2-73dd-40a8-b44b-d8dc20f7d6c8/istio-proxy/0.log"
Nov 03 21:46:07.393433 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:46:07.393401    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-mjcbr_e69b42ce-7354-47e8-ac8f-22aeb8b0a70a/istio-proxy/0.log"
Nov 03 21:46:25.932120 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:46:25.932086    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:46:38.915537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:46:38.915507    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:46:55.932188 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:46:55.932112    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:47:08.667221 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:47:08.667185    5318 scope.go:122] "RemoveContainer" containerID="ba414b45a80107171c63edf61596eca5fd621c21d258341c0a83d1e8a5ff352b"
Nov 03 21:47:25.932474 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:47:25.932437    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:47:39.273011 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:47:39.272978    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:47:39.487460 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:47:39.487427    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:47:55.932851 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:47:55.932810    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:48:25.932393 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:48:25.932290    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:48:29.152216 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:48:29.152183    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:48:29.360636 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:48:29.360605    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:48:39.451915 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:48:39.451882    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nl7nq_7574d083-15a3-4cf9-9547-93c53eb8222a/global-pull-secret-syncer/0.log"
Nov 03 21:48:39.614294 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:48:39.614263    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:48:41.322195 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:48:41.322160    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log"
Nov 03 21:48:43.399786 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:48:43.399760    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-141.us-west-2.compute.internal_ebc5d4efd8a63d505e3fa8dc27ffb1de/haproxy/0.log"
Nov 03 21:48:55.933692 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:48:55.933652    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:49:07.516914 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.516880    5318 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"]
Nov 03 21:49:07.517845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.517464    5318 cpu_manager.go:414] "containerMap: removing container" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerName="cleanup"
Nov 03 21:49:07.517845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.517481    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerName="cleanup"
Nov 03 21:49:07.517845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.517504    5318 cpu_manager.go:414] "containerMap: removing container" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerName="cleanup"
Nov 03 21:49:07.517845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.517512    5318 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerName="cleanup"
Nov 03 21:49:07.517845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.517700    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerName="cleanup"
Nov 03 21:49:07.517845 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.517717    5318 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c57caf-9192-4d08-af4c-c1d07e5a3867" containerName="cleanup"
Nov 03 21:49:07.521287 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.521267    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.569866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.569831    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtnjt\"/\"openshift-service-ca.crt\""
Nov 03 21:49:07.569866 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.569855    5318 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtnjt\"/\"kube-root-ca.crt\""
Nov 03 21:49:07.570112 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.569996    5318 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtnjt\"/\"default-dockercfg-bkvpt\""
Nov 03 21:49:07.618677 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.618653    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"]
Nov 03 21:49:07.651469 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.651442    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-podres\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.651614 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.651513    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-proc\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.651614 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.651601    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-sys\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.651700 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.651630    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8qw\" (UniqueName: \"kubernetes.io/projected/13cd14cb-45e7-4a79-b702-642356aead73-kube-api-access-bn8qw\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.651700 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.651660    5318 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-lib-modules\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753028 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753006    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-proc\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753047    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-sys\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753065    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8qw\" (UniqueName: \"kubernetes.io/projected/13cd14cb-45e7-4a79-b702-642356aead73-kube-api-access-bn8qw\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753090    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-lib-modules\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753111    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-proc\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753128    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-sys\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753165 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753156    5318 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-podres\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753178    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-lib-modules\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.753418 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.753244    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13cd14cb-45e7-4a79-b702-642356aead73-podres\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.810140 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.810081    5318 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8qw\" (UniqueName: \"kubernetes.io/projected/13cd14cb-45e7-4a79-b702-642356aead73-kube-api-access-bn8qw\") pod \"perf-node-gather-daemonset-nqgdw\" (UID: \"13cd14cb-45e7-4a79-b702-642356aead73\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:07.835048 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:07.835027    5318 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:08.196772 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:08.196705    5318 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"]
Nov 03 21:49:08.199170 ip-10-0-135-141 kubenswrapper[5318]: W1103 21:49:08.199141    5318 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13cd14cb_45e7_4a79_b702_642356aead73.slice/crio-996aba065bdda16f931d308e89da65abce0e1419f37d2d96f3af1b29f66bab45 WatchSource:0}: Error finding container 996aba065bdda16f931d308e89da65abce0e1419f37d2d96f3af1b29f66bab45: Status 404 returned error can't find the container with id 996aba065bdda16f931d308e89da65abce0e1419f37d2d96f3af1b29f66bab45
Nov 03 21:49:08.359553 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:08.359520    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw" event={"ID":"13cd14cb-45e7-4a79-b702-642356aead73","Type":"ContainerStarted","Data":"27cd59ce040ae2926858ba0c1770485cfe427dab7200c7f1e406da715df897d4"}
Nov 03 21:49:08.359706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:08.359580    5318 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw" event={"ID":"13cd14cb-45e7-4a79-b702-642356aead73","Type":"ContainerStarted","Data":"996aba065bdda16f931d308e89da65abce0e1419f37d2d96f3af1b29f66bab45"}
Nov 03 21:49:08.359706 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:08.359613    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:08.466979 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:08.466877    5318 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw" podStartSLOduration=1.46686074 podStartE2EDuration="1.46686074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:08.466545098 +0000 UTC m=+4741.446101939" watchObservedRunningTime="2026-11-03 21:49:08.46686074 +0000 UTC m=+4741.446417581"
Nov 03 21:49:08.701672 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:08.701644    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-llhxk_6218182d-10a0-4142-b413-46c95a9ecee8/kuadrant-console-plugin/0.log"
Nov 03 21:49:09.181538 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:09.181509    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-mttnr_d4abe529-55f5-4e84-81df-45f1f7e89746/manager/0.log"
Nov 03 21:49:09.796407 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:09.796373    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-j6xpz_ec3e62c6-7c98-4e0e-ad88-4653f9a1982e/limitador/0.log"
Nov 03 21:49:13.047067 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:13.047039    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/alertmanager/0.log"
Nov 03 21:49:13.467249 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:13.467225    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/config-reloader/0.log"
Nov 03 21:49:13.889109 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:13.889081    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/kube-rbac-proxy-web/0.log"
Nov 03 21:49:14.310442 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:14.310369    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/kube-rbac-proxy/0.log"
Nov 03 21:49:14.382700 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:14.382670    5318 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-nqgdw"
Nov 03 21:49:14.731200 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:14.731171    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/kube-rbac-proxy-metric/0.log"
Nov 03 21:49:15.152879 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:15.152847    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/prom-label-proxy/0.log"
Nov 03 21:49:15.573784 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:15.573717    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f88fddc-5a96-486b-b774-85753bc6c023/init-config-reloader/0.log"
Nov 03 21:49:17.716647 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:17.716616    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-674d9b67d4-m4jmd_0762ba86-bec7-4282-a042-ca63ffaeeea9/metrics-server/0.log"
Nov 03 21:49:18.138916 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:18.138891    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-9gkw6_bcba368e-649b-4dec-a6f3-edfb0f20ba38/monitoring-plugin/0.log"
Nov 03 21:49:19.789211 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:19.789184    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4ksx_1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/node-exporter/0.log"
Nov 03 21:49:20.045174 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:20.045095    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"]
Nov 03 21:49:20.090986 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:20.090951    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ffsl9"]
Nov 03 21:49:20.209029 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:20.209001    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4ksx_1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/kube-rbac-proxy/0.log"
Nov 03 21:49:20.628775 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:20.628750    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4ksx_1af07fe5-98e6-4f65-a70f-1b5cfe9c1735/init-textfile/0.log"
Nov 03 21:49:21.254855 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:21.254824    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b541c92-c60b-4345-a238-7c5bceccc120" path="/var/lib/kubelet/pods/6b541c92-c60b-4345-a238-7c5bceccc120/volumes"
Nov 03 21:49:22.288126 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:22.288090    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jgv9m_298b5083-4edb-4714-96e4-a9d9a9a4a1e1/kube-rbac-proxy-main/0.log"
Nov 03 21:49:22.837905 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:22.837875    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jgv9m_298b5083-4edb-4714-96e4-a9d9a9a4a1e1/kube-rbac-proxy-self/0.log"
Nov 03 21:49:23.256546 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:23.256475    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jgv9m_298b5083-4edb-4714-96e4-a9d9a9a4a1e1/openshift-state-metrics/0.log"
Nov 03 21:49:23.684561 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:23.684529    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/prometheus/0.log"
Nov 03 21:49:24.103748 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:24.103715    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/config-reloader/0.log"
Nov 03 21:49:24.523246 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:24.523173    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/thanos-sidecar/0.log"
Nov 03 21:49:24.943104 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:24.943075    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/kube-rbac-proxy-web/0.log"
Nov 03 21:49:25.363115 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:25.363093    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/kube-rbac-proxy/0.log"
Nov 03 21:49:25.781704 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:25.781631    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/kube-rbac-proxy-thanos/0.log"
Nov 03 21:49:25.933263 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:49:25.933233    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"
Nov 03 21:49:26.203537 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:26.203510    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4df91e9-57b1-42d3-9308-dd68df3223c8/init-config-reloader/0.log"
Nov 03 21:49:26.634734 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:26.634703    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x8c4k_3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/prometheus-operator/0.log"
Nov 03 21:49:27.050115 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:27.050046    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x8c4k_3cae6ab2-e17e-4b19-8e99-f36f7072a3b7/kube-rbac-proxy/0.log"
Nov 03 21:49:29.142875 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:29.142847    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9cd56d8-tsp6l_056b04d8-72b3-4019-bddd-56a4469906de/thanos-query/0.log"
Nov 03 21:49:29.563564 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:29.563481    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9cd56d8-tsp6l_056b04d8-72b3-4019-bddd-56a4469906de/kube-rbac-proxy-web/0.log"
Nov 03 21:49:29.991516 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:29.991486    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9cd56d8-tsp6l_056b04d8-72b3-4019-bddd-56a4469906de/kube-rbac-proxy/0.log"
Nov 03 21:49:30.411829 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:30.411798    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9cd56d8-tsp6l_056b04d8-72b3-4019-bddd-56a4469906de/prom-label-proxy/0.log"
Nov 03 21:49:30.833308 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:30.833229    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9cd56d8-tsp6l_056b04d8-72b3-4019-bddd-56a4469906de/kube-rbac-proxy-rules/0.log"
Nov 03 21:49:31.256175 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:31.256141    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85d9cd56d8-tsp6l_056b04d8-72b3-4019-bddd-56a4469906de/kube-rbac-proxy-metrics/0.log"
Nov 03 21:49:36.949484 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:36.949447    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:49:39.939768 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:39.939733    5318 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k7ks_4bce3daf-4de1-4fea-89e8-cd891e432355/konnectivity-agent/0.log" limit=1024
Nov 03 21:49:41.044254 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:41.044210    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"]
Nov 03 21:49:41.090415 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:41.090382    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-p9lgp"]
Nov 03 21:49:41.254296 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:41.254264    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de886ca3-921c-4662-868e-5005d3fc0ebb" path="/var/lib/kubelet/pods/de886ca3-921c-4662-868e-5005d3fc0ebb/volumes"
Nov 03 21:49:47.551161 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:47.551077    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5vdpt_0305c1b2-ad03-4e5b-847f-64b2828fedc2/dns/0.log"
Nov 03 21:49:47.968016 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:47.967992    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5vdpt_0305c1b2-ad03-4e5b-847f-64b2828fedc2/kube-rbac-proxy/0.log"
Nov 03 21:49:50.831070 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:50.831043    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hk7dd_0be5ee45-4c57-4f1f-a3a1-075c2dca3c90/dns-node-resolver/0.log"
Nov 03 21:49:53.043643 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:53.043607    5318 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"]
Nov 03 21:49:53.089557 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:53.089527    5318 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8bdfq"]
Nov 03 21:49:53.256848 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:53.256804    5318 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47f1745-99da-457d-a6e8-2052c061589d" path="/var/lib/kubelet/pods/d47f1745-99da-457d-a6e8-2052c061589d/volumes"
Nov 03 21:49:54.477284 ip-10-0-135-141 kubenswrapper[5318]: I1103 21:49:54.477256    5318 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kxpt4_a641934f-d459-46af-be40-6aee398f3b82/node-ca/0.log"
Nov 03 21:49:55.934533 ip-10-0-135-141 kubenswrapper[5318]: E1103 21:49:55.934498    5318 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vt4cx" containerName="extractor"